Appendix 2
On the True Nature of Software
This appendix provides the basic elements necessary to understand the specificities, and hence, power of software, as a distinctive activity requiring consideration, management and metrics.
A2.1. Software role, software people role
A washing machine, a car radio or almost any product of everyday life now incorporates some amount of software (often a lot). This software is frequently “buried”, in the sense that its presence is not obvious or “visible”, neither for the end-user nor even for the manager responsible for its development.
In many cases, softwares have creeped into long-lived products – frequently through the man–machine interface – without any noticeable change in the development process.
As time goes by, software development costs demonstrate that software is no longer negligible, but rather may even become the main product.
However, the technical hierarchy within the enterprise remains unchanged. If, for example, the company makes electronic products, the electronics specialists will still occupy all management positions, with no intention of granting software people the smallest part of their influence. A similar situation occurs when other engineering specialities are historically dominant in the design process: hydraulics, mechanics, etc.
Even in companies already confronted with several coexisting engineering disciplines (such as mechanics and electronics, biology and computing or material sciences and chemistry), and therefore, we would think, its better to be prepared for change, software will always be considered as something “apart”, as we had the opportunity to observe throughout enduring professional experience.
The under-representation of software specialists, in places where decisions are taken, unavoidably leads to poor decisions, whether from a technical or managerial standpoint:
- – Technical given the increasing capabilities of programmed solutions which make them good candidates to replace other technologies, and refusing to evolve in this direction is a sign of a product which will be rapidly made obsolete by other, more innovative competitors.
In the business of home regulation systems or temperature measurement devices, long established and powerful players, like Honeywell or Siemens, feel the push of totally new players, of a much smaller size (for example the French startup Netatmo), without any previous presence in the domain, but with products designed to be seamlessly integrated in iOS or Android ecosystems. With the Internet of Things, a massive trend in this direction is underway.
- – Managerial because a company that doesn’t manage to give software people the decision power, in line with their importance with respect to the product, falls into the traps described above.
Software activities are always the last ones to perform. They are indeed impacted by any last minute change made in the hardware first, of which material embodiment they depend for final tests and verifications. Due to this dependency, software people are often under scrutiny from top level management and held responsible for any delay.
Of course, just like any other contributors, they have their share of responsibility. But it is well known that, in large multi-year projects, nobody will question a weeks delay in the early phases, whereas the top level management will require real time reporting for a single half day delay at the end. As it happens too often by that time, their concern will be to identify and punish the culprits (but not necessarily in this sequence), or to suggest false good ideas (such as parallelization up to a ridiculous and counterproductive extent).
In companies where managers have a poor software culture, it is very easy for the non-software contributors to place the blame for delays on software people shoulders, and thereby ruin their reputation.
On the other hand, companies with a good understanding of software issues not only perform well for software, but also take advantage of this knowledge to cross-fertilize other engineering disciplines. For example, software people are good at requirements management, because they are confronted with highly complex products in their line of work.
The expertise gained by software people in adopting a functional standpoint, and in managing and controlling the baselines of each and every associated requirement, is something extremely valuable for other disciplines, which, for historical reasons, are usually behind on these subjects.
The more advanced companies in space, aeronautics or automobile industries have developed a systems engineering discipline, adapted to large, multidisciplinary projects, wherein software appears as one technology among others. Despite this, they mostly fail to grant software people access to top level management.
A2.2. Software, an immaterial product
Software Project Tracking
“The invisible part of everything that you thought you could see, you can’t see.”
G.W. Bush (offering his appraisal of the Palestinian/Israeli situation, Interview with ITN, Crawford, Texas, Apr. 5, 2002)
Software is probably the only engineering discipline where the manager is unable to estimate the progress of work. In software, the well-known reply to the manager asking about work progress is “90% finished”, but can be stuck there for an indefinite period of time.
A software project in industrial conditions consists of managing thousands or, more likely, tens of thousands of artifacts (code, but also various documents, i.e. test procedures, test results, etc.), about which a “software illiterate” manager has no idea at all.
This quantity also rapidly increases with the level of quality required, being maximal for software critical software subject to particular requirements, or checked by independent authorities (EASA in Europe and FAA in the US for the aeronautics case). Of course, in this case, delays and costs explode, as compared with a “normal” development.
Industrial solutions exists to develop software with minimum bugs, but these are costly, even very costly, and, which is even worse, in a business where time to market is essential, they also require much more development time.
It could be argued that such considerations concerning critical software reside far away from Apple’s business, but this could turn to be a short sighted view. What if Apple decided to move toward the automobile industry, with self-driving vehicles, subject to safety critical requirements regulations?
As a way to circumvent the difficulty he/she has to visualize work progress, an incompetent manager grasps the only thing which is accessible to him/her: the mythical number of lines of code. Unfortunately, this “metric” remains more delicate to use than it seems, is programming language dependent, and is in no obvious way correlated to the effort, nor to the work progress.
Generally speaking, sticking to effort estimation alone, only the function points method makes an estimation of the software development effort possible, independently of the coding language used. Unfortunately, it entails some expertise, and requires a detailed functional specification of the software product to be developed, which is beyond most organizations capability.
A manager relying on lines of codes monitoring to track and manage a project will go nowhere as nothing is easier than developing lines of code, especially with modern development methods. Of course, the question of whether these are necessary, and of required level of quality, is another story.
Indubitably, in the absence of other metrics, a rough estimate of the volume of code to be produced makes no harm. Because, at project launch, it is much better to know whether the final product will be in the range of thousands, or the tens of thousands, of lines of code.
As a result, a manager is unable to properly manage a software project without a good understanding of the very nature of the software activity. And no MBA will help the manager in this regard.
Project planning
“Without music to decorate it, time is just a bunch of boring production deadlines or dates by which bills must be paid.”
Frank Zappa
Project management, planning and scheduling, are now firmly established disciplines, based on a large set of good practices and knowledge, and this is not the place to recall them in detail.
Let us just go back to the basics. A “good” schedule is one obtained after all tasks have been identified (Work Breakdown Structure, or WBS), estimated (in effort first, and then in duration, given resources hypotheses), and later on nested one to another according to the sequencing (and possibly, parallelization) constraints. Dedicated software applications easily calculate the earliest delivery date possible, after all these elements are estimated and introduced.
If the resulting output isn’t in line with the objective, hypotheses are modified but always in a forward calculation mode.
Needless to say, a backwards calculation mode, consisting of going back in time without any feasibility consideration, from the specified due date to the current time, is not a scheduling exercise, but nothing more than disguising the manager’s fantasy into some fake reality. Of course, a manager can always twist the software project leader’s arm, possibly using authority, but in this case putting him/herself beyond the notion of commitment, as a result of a negotiation; a hard one possibly, but nevertheless a negotiation.
All the foregoing about scheduling is true, whatever the domain considered. The software specificity is that, if the manager exerts an unrealistic pressure on delays, he or she will get due value for money. It is always possible to deliver a piece of software at any given point in time, but too early and it could be functionally incomplete, or poorly tested, or both.
We have no information on how Apple schedules its developments. We would need to wait for iPad or iPhone developers to write their memoirs to have some indication on this.
As briefly mentioned later, manager “Steve Jobs version 1.0” was not reputed to base his schedules on the virtuous approach described before.
However, a “Steve Jobs 2.0”, and more generally Apple now, seemed to adopt a more realistic attitude on this topic.
Of course, the launch of the Maps Application (the Google Maps competitor) can be presented as a notable exception, but the trouble was more in the data than in the code itself, and mapping the entire planet surface, down to the smallest lane, isn’t easy. Assuredly, the mistake was more in not positioning the first version as a Beta version, which was the only realistic possibility at that time.
This episode resulted in the departure of Scott Forstall (then Apple’s iOS Senior Vice President). Even though other reasons probably explain his mistakes, successors are warned of what their fate will be in case their management leads to a similar failure.
This does not mean (and we come back to this later) that the Apple developers do not feel a huge pressure on their shoulders, but this does not go as far as requiring the impossible from them, without listening to what they say. That being said, in the present case, an irresponsible attitude, which characterizes a contempt of software people and software in general, can lead to nothing but disaster.
A2.3. Software development activities – the CMM model
“If you don’t know where you’re going, you could wind up somewhere else.”
Attributed to Yogi Berra,
American baseball player
The CMM model
A manager without a software culture identifies software development as coding. This couldn’t be further from the truth, as we will see through the CMM model developed by former IBM Vice President Watts Humphrey at Carnegie Mellon University-based SEI (Software Engineering Institute).
The capability maturity model (CMM), initially dedicated to software (or its SPICE, or software process improvement and capability determination (ISO/CEI 15504) equivalent), evolved into capability maturity model integration (CMMI), a more general model, extended to any engineering activity, including mechanics, electronics, etc.
In what follows, we will stick to the CMM model, because, although partly obsolete, it is simpler and dedicated to software, which makes it more suitable for our purpose.
Any organization developing software can be situated at one of the five levels of the CMM model, described hereafter. From the initial level 1 where most companies are, process improvement initiatives aim at reaching higher levels, and possibly the ultimate objective of level 5. This level is hard to reach but not impossible, and has been reached by a number of organizations (many of them in India).
“Initial” (maturity level 1)
No company standard process exists at this level. Weaknesses are not identified and projects participants are not precisely informed of what is expected from them. Reaction to problems happen in panic mode, without a clear identification of priorities. Still, it may happen that, due to the exceptional talent and commitment of the project leader, some projects can be successful, but this is just a result of heroic behavior, which, on the long term, always leads to discouragement or burn out but rarely to the next level:
“Managed” (maturity level 2)
Each project follows a disciplined approach, as documented in several plans (development plan, quality assurance plan, configuration management plan, etc.). At this level, it is the project leader’s responsibility to define, apply and maintain the corresponding plans.
“Defined” (maturity level 3)
At this level, a company-wide discipline is documented and applied, through a standardization and inter-projects capitalization. The project plans result from instantiating the generic plans. The developers are adequately trained.
“Quantitatively managed” (maturity level 4)
Projects are managed on the basis of quantitative measurable objectives, related to both products and processes. The quality objectives expressed by the customer are translated into project objectives, and form the basis of the project planning.
“Optimizing” (maturity level 5)
All processes (including the improvement process) are predictable and quantitatively managed.
At this level, which is a sort of developers nirvana, everything, absolutely everything, is planned and happens according to plan in absolute calm and serenity, without a need for the single hour of overtime1.
This certainly isn’t the working environment of the Macintosh team (see Andy Hertzfeld’s book [HER 04]). Moreover, incompetent management prefer level 1 Brownian activities, which they assimilate to efficiency.
The structure of the CMM model distinguishes several key process areas, the 18 KPAs, associated with one of the five maturity levels of the model (Table A2.1).
Table A2.1. The 18 key process areas of the original CMM model
Every key process area can be further developed into key practices and the level of implementation indicates whether the key process area can be considered as satisfied or not.
For example, requirements management (RM) includes 12, project planning (PP) has 25, etc. In total, 316 practices are associated with the 18 key process areas of the CMM model.
Reaching a given maturity level requires that all key process areas associated with this level are fully implemented.
The fact that no key process area is associated with level 1 does not mean that people at this level just do nothing. To the contrary, level 1 is characterized as hectic and inefficient because it can be disorganized.
Key process areas can be present at any maturity level. Even at level 1, it is possible to find practices related to defect prevention (DP), which is normally a level 5 key process area. Similarly, software quality assurance (SQA) does not, obviously, start at level 4.
What the model says is that it would be counterproductive to place the emphasis of the improvement efforts on key process areas which are not associated with the targeted level: this will come in due time.
Of course, nothing more than guesses can be made about the level of maturity of Apple development teams. Apple does not communicate on this.
It is unlikely that Apple has internally set software maturity objectives to reach, because companies reaching high levels of maturity do communicate on this, as a legitimate source for pride.
It can be, however, taken for granted that a manager at Apple is not remaining with eyes wide open when asked for resources to perform requirements management, configuration management, quality assurance, verification tasks, etc.
In a very remarkable way, and extremely shocking at first sight, it can be noticed that there is no such thing in the CMM model as a key process area, or even activity, related to coding.
From the model standpoint, this does not mean that the coding activity does not exist, but rather that it is not considered as a key one, on which it could be possible to distinguish a non-mature organization from a mature one, or which could be related to project failures.
Even though a manager may not subscribe to a software process improvement objective – after all, this is an own decision, and we will come back to it later – he must know and understand the particular nature of a software development, and a model like the CMM is suitable for it, as a sort of functional analysis of everything which has to be performed, irrespective of the particular development environment, or methodological choices, of the company.
The mystery of the small, yet costly software fix
“Ces choses lá sont rudes. Il faut, pour les comprendre, avoir fait des études". (Those things are tough; understanding them requires a degree)
Victor Hugo
La légende des siècles (1859)
The following true story is a good illustration of the common misunderstandings of software, which is too frequent among managers.
Further to a problem of software origin encountered in the field, a software fix had been defined, likely as simple as modifying a few instructions in the code (sometimes one single instruction suffices). We may note this also happens frequently for problems without a software cause, but which can be fixed by software.
Everything being fine, the manager expected the fix to be implemented at once, and of course, at virtually no cost. In the end, he was unable to understand why the software teams in charge submitted him a substantial cost and delay estimation, if not an obvious lack of motivation on their side.
But this doesn’t have to be the case. Unless authorizing dangerous and uncontrollable modifications on the spot, any software change must undergo a precise sequence of steps, impacting a considerable number of software artifacts, which have to be managed and controlled in a very specific way, not to mention the necessary verification and non-regression tests, etc.
With complex software, this cannot be just a one person task.
A manager who refuses to make the intellectual effort to understand this reality will have a hard time with his software teams. The loss of confidence will translate into conflicts, and possibly, subtle retaliation strategies from the software team.
Consultants trying to sell the “process message” had such a hard time that, since 2000, they now prefer to promote “agile” methodologies (including, for example, the SCRUM method). These methodologies complement, rather than contradict, the process approach, as they restrict their field to planning and implementing cycles of development (the iterations), without touching on the much more difficult issue of the programmers organization of work.
A2.4. Software people productivity
“The system is that there is no system. That doesn’t mean we don’t have process. Apple is a very disciplined company, and we have great processes. But that’s not what it’s about. Process makes you more efficient.”
Steve Jobs, BusinessWeek, Oct. 12, 2004
We once had the opportunity to discuss with a CEO employing software people, the concepts of maturity models, process capability, etc.
At this point, he expressed total skepticism toward the potential benefits of the concepts, preferring his own approach, of biblical simplicity. He said:
Given the fact that variations of productivity among programmers can easily range on a 1 to 10 scale, firing the bad programmers and only retaining the best ones, brings a dramatic improvement to global productivity!
The practical difficulty of implementing such a policy, in some environments, is not sufficient to disregard this idea without further analysis.
The statement concerning the productivity variations is not totally false, but it is first of all necessary to explicit the concept of productivity in this case, as well as going back to the root causes.
As already pointed out, productivity under the coding speed angle is meaningless, given the small weight of coding activity within the entire software effort.
In the 1980s, organizing competitions among programmers using different languages or development environments was a common practice, and it led to see which one would be the “fastest” guy. We have seen this in CAD-CAM shows, where the winner was the first programmer to produce the NC tape to machine a given part, defined through its drawing (“blueprint”).
But more than a way of comparing CAD-CAM environments used, this competition was in fact highlighting the dexterity of the winner, his ability to read a drawing, etc.
The capacity to code fast does not appear to us as a very significant issue. However, it is true that some programmers have a distinctive talent, compared to their colleagues, to produce a code which is more easily readable, maintainable, testable, etc.
Process gurus struggle to admit this, as their approach is based on the premises that individuals can be easily substituted, with the same repeatable result, as long as the process is mastered.
Pushing the reasoning to the extreme would lead to fixing the exact same salary for all programmers, which is seldom seen, in practice. Some process ayatollahs go as far as describing a world wherein developers would just have to blindly stick to predefined job instructions, down to the smallest level of detail. Such a vision might possibly be useful in a mass production environment, but certainly not in design environments.
Software productivity discrepancies among individuals can be related, to some extent, to dissimilarities in levels of training or experience, but the explanation falls short of explaining everything.
Undeniably, some individuals are off the scale when it comes to programming, and are to software programming what Michelangelo is to painting or sculpture. We have previously mentioned Jean-Marie Hullot, hired by Steve Jobs, as an example of individual belonging to this very small “population”.