When I get a systems engineer, I don’t know what I’m supposed to be getting.

– Anonymous Helix program manager participant

Chapter 4 explained the proficiencies that systems engineers need to perform their jobs well. This chapter presents a method to visualize how strong someone is in each proficiency – a PROFICIENCY PROFILE – or just PROFILE, for short. We begin with three fundamental truths:

• Virtually no one excels in all proficiencies.
• Depending on the roles being played, strength in some proficiencies is more important than strength in others.
• Often, it is a team of systems engineers who collectively possess the necessary proficiency strength to complete a task, not a single individual.

These truths drive the importance of PROFILES and several distinct uses for them:

1. Personal PROFILE. Created for an individual systems engineer, it can be used as a self‐assessment tool to better understand personal strengths and weaknesses.
2. Validated PROFILE. Managers or peers can validate a personal PROFILE. The validated version can be a basis for career counseling and guidance, helping a systems engineer seek the best education, training, and mentoring to support current and future roles and positions.
3. Screening PROFILE. A candidate applying for a position can complete a profile, which can then be used as a screening tool for a discussion during interviews.
4. Exemplar PROFILE. Leaders at any level, charged with growing some portion of the systems engineering workforce, can select exemplar PROFILES of their best people and share them with others. Exemplar PROFILES show the high level of proficiency that the most effective systems engineers in the organization have achieved.
5. Recommended PROFILE. Similar to an exemplar PROFILE, a recommended PROFILE shows what a manager wants from someone occupying a key position or role without making explicit reference to the PROFILE of a real person.
6. Aggregate PROFILE. Aggregating PROFILES of a group will reveal their collective characteristics, such as the mean of specific proficiencies. Because there is inherent subjectivity in how people self‐assess, an aggregate PROFILE must be used carefully but can still help identify group strengths and weaknesses.

An illustrative personal PROFILE is shown in Figure 6.1 for Matt, the hypothetical senior systems engineer introduced at the end of Chapter 3. Recall that Matt’s career spans 10 years with a very rapid rise – 5 years at DoesThatHurt as a biomedical engineer followed by 5 years at MedBright, first as a systems engineer and eventually as lead systems engineer. Figure 6.1 is a radar or spider plot that has six spokes, one for each proficiency area. Matt’s strength in each proficiency area is marked by a small circle along the radial axis from 1 (least proficient) to 5 (most proficient).

The PROFILE reveals that Matt believes himself to be a strong systems engineer. He self‐assesses with a score of 3, 4, or 5 on every proficiency area using the rubric that will be explained in the next section.

There is nothing sacred about a 5‐point scale to measure proficiency strength. In fact, at different times in our research, the scale has varied from a low of 3 points to a high of 11 points. For purposes of exploring PROFILES in this book, a 5‐point scale is sufficient, but a person or organization could choose a more or less granular scale.

This relatively simple radar diagram shown in Figure 6.1 is easy to complete and understand. The disadvantage is that, with only six data points, it is a coarse view of a systems engineer’s strengths and weaknesses. Nevertheless, for a top‐level look, the more than 100 people who completed these PROFILES as part of our interviews found it insightful.

We can easily imagine expanding Matt’s personal PROFILE in two ways:

• Growing from a single personal PROFILE to a series of them reflecting different points in Matt’s career, e.g. at graduation with his bachelor’s degree in bioengineering, when he completed his time at DoesThatHurt and began working for MedBright, and today. Figure 6.2 shows such a sequence.
• Decomposing the six aggregate proficiency areas into their constituent categories or even topics, creating a much richer tapestry of Matt’s strengths and weaknesses. Figure 6.3 shows an example decomposition down to categories for Matt’s current PROFILE. There are now six radar charts. Each chart represents a single proficiency area broken down into its constituent categories.

The sequence in Figure 6.2 shows that over the 10 years of Matt’s career, his TECHNICAL LEADERSHIP, INTERPERSONAL SKILLS, SYSTEMS MINDSET, and SYSTEMS ENGINEERING DISCIPLINE strength steadily grows. His SCIENCE/MATH/GENERAL ENGINEERING strength remains constant, and his SYSTEM’S DOMAIN AND OPERATIONAL CONTEXT strength grows slightly. In the next chapter, such a sequence of PROFILES will be explored further, first in the context of the forces that lead to such changes in strength and second as part of a larger depiction of a career path – the specific sequence of positions a systems engineer assumes over time together with the corresponding roles, events, PROFILES, and other milestones of consequence in a person’s career.

The sequence of six radar diagrams in Figure 6.3 shows the basis for Matt’s self‐assessment. For example, there are six constituent categories in MATH/SCIENCE/GENERAL ENGINEERING. The individual scores roll up to an overall score of 4. The method used in Helix to calculate a proficiency area score from its constituent category scores, is to compute the average and round off to a whole number. In Matt’s example, he scored 5 in NATURAL SCIENCE FOUNDATIONS and ENGINEERING FUNDAMENTALS, in SOCIAL SCIENCE FOUNDATIONS and PROBABILITY AND STATISTICS scored 3, while he assessed himself as a 4 in CALCULUS AND ANALYTIC GEOMETRY and COMPUTING FUNDAMENTALS. The average of those six scores is 24/6 = 4. Certainly, more sophisticated roll‐up methods are possible, such as to assign weights to individual categories to reflect their relative importance to the organization or to avoid rounding and use noninteger values. Such methods are not shown here, although some organizations who participated in Helix preferred these more sophisticated approaches. For an individual, the most basic assessment using averages is easy and insightful. For an organization, any approach could work as long as it is applied consistently across all individuals.

ASSESSING PROFICIENCY

During organizational interviews, more than 100 people created their own personal PROFILES at the proficiency area level. This section explains how those self‐assessments were conducted and then presents a specific rubric for self‐assessment based on what we learned from those interviews. Small groups of two to seven people from the same organization usually sat together with members of our research team. Figure 6.4 shows an actual worksheet used for some data collection, with a 6‐point scoring scale from 0 to 5. Participants were asked to be mindful of the categories constituting each area but were not asked to score themselves against each category as shown in Figure 6.4. Although there was some variation, participants typically had only 10 minutes to complete a self‐assessment, making it impossible to do a category‐level assessment.

Because we wanted to understand how their proficiency strength changed over time, they created two PROFILES – one as they judged themselves on the day they conducted the exercise and one as they remembered themselves at some point in the past. This sequence is an abbreviated version of the one shown in Figure 6.2, with just two PROFILES rather than three.

We explored several options on how far back in the past a systems engineer’s first PROFILE should be. For some, it was immediately after completing their bachelor’s degree. For others, it was 10 years before the interview or when they joined their current organization. Completion of undergraduate education was a frequent point of comparison, but particularly for those with many years of experience, they were uncomfortable with self‐assessment that far back because they doubted how well they could accurately recall their strength. For each assessment, we recorded the general dates for the point of comparison.

Participants were generally told that a score of 1 meant “only has awareness” and the highest value (usually a number between 5 and 10) meant “among the best in the organization.” We avoided being too prescriptive and told people to use their “gut” to decide where to place themselves on the scale. The rubric itself varied somewhat over time from organization to organization, but this description fairly reflects the general approach.

After everyone in the room completed their PROFILES, the group talked openly about the scores and why they assessed themselves as they did. These conversations moved the group closer to a shared way to score. There were often comments such as “Well, I was going to rate myself a 5, but if you consider yourself a 5, I should be lower” or “I think you’ve under‐rated yourself, I personally see you as ….” Such shared understanding was valuable not only to our research team but for the subjects as well and somewhat reflected what we expected in personal PROFILES. Most people interviewed indicated that the experience of self‐assessing their proficiencies, as subjective as it was, offered insight into their own strengths and weaknesses and how they compared with their colleagues.

Initially, all PROFILE data was entered on paper. This was soon replaced by an Excel spreadsheet that automatically built the PROFILE visualization from entered data. As this book is being written, a Web‐based tool for data collection, analysis, and visualization is being developed.

EXEMPLAR AND RECOMMENDED PROFICIENCY PROFILES

The best people in an organization can help their colleagues grow by publishing their PROFILES as exemplars, i.e. profiles of leading systems engineers together with a rationale from those individuals on why their profiles look as they do. An exemplar PROFILE serves as the North Star for others who aspire to achieve their success. Someone entering the organization and looking to understand “what success looks like” could examine PROFILES of the best for guidance. Of course, the availability of this information will depend on personnel policies, organizational culture, and the willingness of people to share information about themselves. Consider the situation where a suitable exemplar PROFILE is not available, perhaps because one of the people best qualified to be an exemplar does not wish to share their PROFILE. The manager can develop recommended PROFILES based on hypothetical team members rather than real ones.

In one organization we visited, we performed such an exercise on several positions, including “system analyst.” In this organization, “system analyst” is a specific position that included the systems engineering role of SYSTEM ANALYST. Their management recommended that an individual be “4/Advanced” in every proficiency area. We then interviewed a number of their system analysts who self‐assessed their personal PROFILES to understand how well they stacked up compared to management recommendations. Most fell short, in some cases by quite a bit.

There are a few things we learned by running this exercise with this organization and others. It is a common and very human trait to answer the question “How good do you want an employee to be?” with the answer “the best.” However, it is impossible for every single individual to be the best at everything. In fact, some individuals we interviewed found these “superhero” profiles to be discouraging. If the organization truly expects them to be at least “Advanced” strength in every proficiency area, they felt they would always fall short. Among very junior systems engineers, some felt that it would take 10–15 years to grow to that level, which is fine provided management believes that “4/Advanced” everywhere is a long‐term goal and not an immediate requirement to fill the system analyst position.

AGGREGATE PROFICIENCY PROFILES

The complete set of an organization’s or a team’s personal PROFILES can be a powerful tool to understand their overall relative strengths and weaknesses. This set becomes even more powerful if they are validated by managers and peers. Simple statistical analyses can be done, calculating, for example, the mean score for specific proficiencies of people who occupy various positions, such as system analyst, system architect, or requirements engineer. Of course, the 5‐level scale is only ordinal. There is clearly some subjectivity in how scores are assigned, even when using the proficiency scale in Table 6.1. Hence, it is important to recognize the limitations of such statistics. Nevertheless, they can be most helpful in understanding the general characteristics and patterns of strength and weakness in an organization. That understanding can guide workforce investments and policies.

Consider the organization discussed above that expected every system analyst to be “4/Advanced” in every proficiency area. We developed an aggregate profile for those we interviewed as shown in Figure 6.5, allowing for noninteger average values. One could read Figure 6.5 to mean that many system analysts do not meet management expectations and therefore are not effective. However, Figure 6.5 could also show where the workforce needs to grow over time, which could drive investment in training, rotational assignments, and other steps to improve the aggregate PROFILE. Also, Figure 6.5 groups all system analysts into one position. That grouping could be refined into three PROFILES, reflecting junior, mid‐level, and senior system analysts, for which different recommended PROFILES would be appropriate. Setting proper expectations is vital.

NOTES AND REFERENCES

1. 1 NIH (n.d.). Competencies Proficiency Scale. National Institute of Health. https://hr.nih.gov/working‐nih/competencies/competencies‐proficiency‐scale (accessed 10 December 2017).
2. 2 Sedlacek, W.E. (2004). Beyond the Big Test: Non‐Cognitive Assessment in Higher Education, 1e. San Francisco, CA: Jossey‐Bass.
3. 3 Sedlacek, W.E. (2005). The case for noncognitive measures. In: Choosing Students: Higher Education Admission Tools for the 21st Century (ed. W. Camara and E. Kimmel), 177–193. Mahwah, NJ: Lawrence Erlbaum.
4. 4 Sedlacek, W.E. (2010). Noncognitive measures for higher education admissions. In: International Encyclopedia of Education, 3e (ed. B. McGaw, E. Baker and P.L. Peterson), 1–10. Amsterdam: Elsevier.
5. 5 Sedlacek, W.E. and Adams‐Gaston, J. (1992). Predicting the academic success of student‐athletes using SAT and noncognitive variables. Journal of Counseling and Development 70: 724–727.
6. 6 MITRE (n.d.). MITRE corporate overview: our history. https://www.mitre.org/about/our‐history (accessed 3 December 2017).
7. 7 MITRE (2017). MITRE Annual Report 2016. https://www.mitre.org/sites/default/files/publications/MITRE‐annual‐report‐2016.pdf (accessed 3 December 2017).