Collecting and Analyzing Coaching Data

• why collecting data is vital for coaching success
• the three primary sources for gathering data
• how to choose the right data-gathering tool for the job
• the importance of data analysis
• how to organize data to get the best results from your client

Introduction: Jocelyn’s Case

Jocelyn is a long-tenured, senior director in a large association. Four months ago, she contacted a coach. During their initial meeting, Jocelyn explained that she wanted to focus on prioritizing her projects and balancing her time. She said that many people made demands of her and that she felt as though she wasn’t meeting those demands as expected. One of the questions her coach asked her was what commitments or projects might she be willing to delegate or let go of for a time, to which Jocelyn replied, “Oh, there isn’t anything I can give up.” When her coach asked her, “What is the reward you are currently realizing for approaching your projects and commitments in the way you do?” Jocelyn was stumped. She said that she really didn’t know and that she’d have to think about the answer. Although Jocelyn and her coach were supposed to meet two weeks later, they did not meet again for three months. Jocelyn’s schedule prevented her from making the meetings as agreed.

When Jocelyn and her coach met next, Jocelyn shared that her boss had asked her what progress she was making in coaching. Jocelyn’s boss told Jocelyn that she was concerned with her schedule, her attendance at meetings, and the way she was meeting or missing her deadlines. In response to a suggestion from her coach, Jocelyn agreed to complete a purpose statement exercise and a self-report on values. She also agreed that some data gathering would help her get a picture of how she was being experienced by others, and she selected several direct reports, peers, others, and her boss for her coach to speak with. Her coach spoke with everyone Jocelyn had on her list. The data revealed that people were conflicted about Jocelyn. On the one hand, they recognized that she had great knowledge and, if they could capture her attention, she was willing to share that knowledge and insight. Among Jocelyn’s strengths, people noted the following:

• upside potential
• innate capability
• intelligence
• resourcefulness
• knowledge
• diligence.

On the other hand, she was frequently running to scheduled meetings, arriving late, seemingly distracted, and leaving early to get to yet another meeting. People wondered about that behavior. She seemed disorganized, chronically late, and out of control. They wondered which Jocelyn was going to show up at any given time. Among the areas to strengthen, people noted the following:

• manage time in such a way to meet deadlines, get to meetings on time, and eliminate chronic lateness
• make better judgments about her job duties so she did not end up doing something that was not really her job and not involve the people whose job it was
• prioritize her job duties so that everything would be completed
• eliminate the perception that she was in it for herself.

Jocelyn’s coach decided to present the data she gathered in the two broad categories noted above. She used specific verbatim comments Jocelyn’s raters shared with her as had been agreed by Jocelyn and her raters. Jocelyn listened to the feedback, and in the conversations that followed, she and her coach worked through the potential upside of re-evaluating her schedule and the potential downside of continuing the status quo. Her coach tested out several hypotheses and helped Jocelyn think through the messages in the feedback. She also shared (with Jocelyn’s permission) her own observations and impressions based on their time working together.

Jocelyn’s case reveals some of the opportunities and challenges in collecting and analyzing data. During their coaching conversations, Jocelyn’s coach purposefully emphasized her strengths. As Marcus Buckingham describes in Go Put Your Strengths to Work, “the radical idea at the core of the strengths movement is that excellence is not the opposite of failure, and that, as such you will learn little about excellence from studying failure” (2007, 5). In other words, “a person or an organization will excel only by amplifying strengths, never by simply fixing weaknesses” (8).

The implication for coaching is substantial. Gathering data on strengths and successes and identifying winning strategies will provide the greatest leverage for growth and development. This is a departure from the more traditional approach, which includes looking at weaknesses. Depending on your clients and their organizations and the degree to which they have embraced the strengths movement, you will likely tailor your approach in gathering and formatting data. It is our experience that in the interests of full disclosure, the clients deserve to hear feedback on what is working as they intended and what is not. During the coaching process, you will be able to support clients in focusing and building on their existing strengths, help clients redirect behaviors that are not serving them, work with clients to develop desired skill areas, and help clients add skill sets and competencies. As shown in figure 5-1, this step in the model is presented as a content step because of its clear focus on data.

Typically, information is out there. The challenge is in capturing it and packaging it in such a way that it is of use to your clients. This chapter explores the sources and methods for gathering data at the individual, team, and organizational levels. The link among beliefs, behaviors, and results is explored as a way to frame data, manage cognitive dissonance, and guide change. Specific topics we will examine in this chapter include reasons to collect data, data sources, methods, and tools to use in collecting data; the pros and cons of qualitative versus quantitative data; logistical considerations in collecting data; considerations in analyzing data; and approaches to extracting themes from data and forming hypotheses when analyzing data.

Data Collection

Data are an integral part of effective coaching. The benefits of collecting data influence individuals, teams, and entire organizations. Some of the benefits of collecting and feeding back data include

• aligning intentions (clients’) with perceptions (others’)
• comparing the where we are with the where we want to be
• demonstrating openness and accountability
• supporting continuous learning
• helping create a high-performance culture.

When choosing a data source and method, it is important to think of the kind of data needed and how the data will be used. Once information is collected, it needs to be sorted into meaningful themes, which can be easily heard and understood by the receiver. Information is powerful when it is used as a way to seek continuous improvement.

Data can be collected from three primary sources.

First, you can collect information directly from your clients. This works whether the client is an individual, a team, or an organization. You can pose a series of questions and collect and analyze the answers. You can review documents that clients have created (for example, emails, reports, policies, meeting minutes). You can take a retrospective focus, analyzing past performance and behavior, and you can take a prospective focus, asking clients to define their desired results and identifying the necessary performance and behavior required to achieve those results. The inherent limitation of this data source (the clients) is the singular perspective they bring. Clients look at themselves, and others, through their own lens, which only represents a part of the picture. They may also have a vested interest in presenting themselves in the best possible light, and as a result they may color or shade information they share.

Second, you can collect information from others. Others may include co-workers, a boss, customers, direct reports, and family members. Others are anyone in addition to the client who is in a position to comment knowledgeably on the subject areas of focus. Others may have a concern about confidentiality and the intention of the data-gathering process. In addition, credibility, motivation, and intent have to be assessed and taken into consideration when collecting data from others. The benefit of others as a data source is that you are likely to collect many parts of the picture of the individual, team, or organization. The chances that you will gather a more complete picture are greater, and you can create the opportunity for your client to enroll these others in his or her future development and success. By including others in the data-gathering part of the coaching process, you introduce them to a role they might play going forward.

Third, you can collect information from yourself, the coach. In your work with individuals, teams, or organizations, you will have the opportunity to observe their behaviors and interact with them in a variety of settings over time. You can identify behaviors and patterns, note consistencies and inconsistencies, and as part of your agreement with the clients, you can bring this information to their attention. Through the use of skillful dialogue, you can inquire about your observations, test assumptions, and support your clients as they think through the implications of the data you have shared. As Mary Beth O’Neill describes in Executive Coaching With Backbone and Heart, it is likely that you will “notice a relationship between what the [client] talks about ‘out there’ and what actually happens in the moment between the two of you. When you notice this parallel occurrence, you can report your experience. . . and help your client get a clearer picture of what happens with her in her organization” (2000, 33).

Methods and Tools

Whether you choose only one of the data sources above or a combination of sources, you will want to use a data-gathering method or tool that supports your client’s goals. Over time, you will likely collect a variety of tools that you have found work best in different circumstances. All tools have strengths and weaknesses. It is up to you to consider all of your options and to select the best tool for the job. Below you will find a discussion of six popular types of data-gathering tools and their strengths and weaknesses. Specific examples of these tools can be found in table 5-1.

Self-Assessment Inventories

Self-assessment inventories are popular and easy to use. Most are available in hardcopy or online versions, and they are generally inexpensive when compared with other data-gathering tools. In many cases, the clients have access to their results immediately. Many inventories are available in individual and team versions. Some self-assessment inventories are linked to substantive databases, which allow results to be compared with a general population norm or a specific management population. This approach provides a combination data source (self and others) because the clients are completing the assessment on themselves, and they are viewing their results against a larger database. Often this comparison provokes reflection and allows clients to consider where they fall on a given continuum and how they can leverage their strengths.

360-Degree Surveys

The 360-degree surveys are designed to provide individuals or teams with perspectives broader than their own. Most are available in hard-copy or online versions. The surveys have a unit cost that is typically higher than self-assessment products, and they may also require a set-up fee. However, 360-degree surveys provide very substantive feedback and a more complete picture than self-assessments. In addition to receiving their own ratings on a group of competencies or skills, individuals or teams can also receive data (including narrative comments) from the boss, superiors, peers, direct reports, clients, and family members. External administration may be required, and often deadlines have to be extended to allow a sufficient number of raters to respond. Many organizations use 360-degree surveys to support ongoing leadership development, and they publish aggregate results to show how the organizations are progressing on stated areas of interest. The 360-degree surveys can provide very rich, detailed data and are another combination data source.

Image Studies

Image studies are qualitative 360-degree surveys, designed to collect multifaceted pictures of individuals or teams. Interviews are conducted with colleagues, superiors, clients, direct reports, and other appropriate stakeholders for the purpose of identifying how the given individuals or teams are perceived and what their effects on others are. The data, once collected, are sorted into themes and then presented to the client. Verbatim comments (all of them or a representative sample) are frequently included. Image studies are best conducted by skilled coaches who can formulate targeted questions and then hold productive conversations with the selected raters.

Climate and Employee Opinion Surveys

Climate and employee opinion surveys are popular tools for taking the pulse of a department, a division, or an entire organization. Most frequently provided online, these surveys inquire about employees’ satisfaction with a number of specified organization dimensions. They can provide valuable baseline data, are generally easy to use, and can measure progress as organizations work on specific areas. The risk with this tool as with others is in failing to act. If the employees perceive that their feedback isn’t valued or acted upon, the organization’s credibility can suffer.

Focus Groups

Focus groups have long been a popular method of gathering data on specific topics. They require skillful facilitation, and traditional focus groups also require compilation of data. They can provide real-time feedback on current topics of interest to the team or organization. In addition to traditional focus groups, there are many groupware and group systems products that allow you to collect data from individuals and preserve the confidentiality of the source. Following an agreed-upon protocol, people respond to questions using computers or keypads and the results are displayed in aggregate, as percentages, or specifically, with no attribution. Questions can be edited or changed in real time by the facilitator depending on the feedback or response from the group, and reports can be formatted in a variety of ways. These products offer the confidentiality of an online survey and the real-time feel of a focus group. The software also supports brainstorming, decision making, action planning, and consensus building.

Table 5-1. Data Collection Methods.

Observation and Shadowing

Observation and shadowing are most often structured activities where coaches accompany clients as they go about their work activities. The opportunity to see your clients in action is substantial, and, depending on the agreement, coaches can provide real-time feedback on the behaviors they experience or they can provide feedback later in a planned coaching session. Some organizations may not welcome this type of data gathering, and not all clients are comfortable granting this level of access. A less structured example of observation can occur with the coach providing observations to the client during their coaching meetings.

Making a Choice

Given the wide range of data-gathering tools available, your final selection will likely be determined by the answers to the following questions.

What tools offer the best fit with your clients? In other words, what sort of data will the clients find credible? Do they or their organization prefer narrative or statistical data? What comfort level do they have with your direct access to data sources? What is their learning style, and what tool will best support it? Fit is defined by your clients and their organizations. You want to select a tool that will be meaningful and an approach that will support, not weaken, the coaching process.

What are the available resources? Specifically, how much time is available for data gathering and analysis? What is the budget for purchasing tools? What kind of time are raters willing and able to spend participating in the process? You want to select a tool that considers the resources you have to work with and will give you the best balance of depth and breadth in your data collection.

What is the intended use of the data? Of course, the primary purpose of gathering data is to provide your client with information to support action. In addition, what is the interest in resurveying the same raters at a later date? How important is it to provide pre- and post-coaching data? What is the benefit of using a tool that will allow a team composite report in addition to an individual report? As the coach, you want to select a tool that supports the measures of success outlined and is agreed-upon by your clients and your clients’ organizations.

Consider the following scenarios, and apply the questions above. Where do your answers lead you? What data-gathering tools would you recommend in each case? Why?

1. Scenario: The employees in a program office have had six directors in six years. They complete an online survey annually and have indicated a lack of trust in senior management. In addition, communication, recognition, respect, and career development received negative ratings. The new director joined the office just following the most recent survey, and he is concerned with the ratings. He wants to get clarification on several content areas, and he wants to address the areas of most concern.
   
   Answer: You may be thinking of electronic focus groups or one-on-one interviews as your preferred data-gathering tools. Both of those choices will allow you to ask clarifying questions and collect data while preserving confidentiality. There is also a personal dimension to those tools that may support the effort whereas another survey may not.
   
   
2. Scenario: An executive and her team of eight vice presidents want to know how they are perceived as the top team of the organization, specifically as related to aligned communications and decision making. They want to get detailed feedback on behaviors with specific examples so they can work on real changes. They are very numbers driven and want to have a baseline against which to assess their progress.
   
   Answer: A 360-degree survey might suit the needs of the executive team by providing detailed feedback and a baseline of information from which they can build. Many 360 tools allow for verbatim comments, which can further detail what people are experiencing. In addition to the individual reports, many tools can also generate team reports so they can look at trends, strengths, and areas to develop as a team. Because they are numbers driven, they may trust a 360 with substantial quantitative content more than a tool that generates only qualitative data.
   
   
3. Scenario: A manager is experiencing a lot of resistance and complaints from her staff. She is very helpful to colleagues and is very responsive to her boss. She doesn’t understand what is causing her to be ineffective with her staff. After all, she was one of them before she was recently promoted to her current position.
   
   Answer: Your client may be facing a skill issue or an interpersonal or cultural issue since she has moved from being one of the team to being the boss. One-on-one interviews may provide multiple perspectives from her staff and allow for clarifying questions. You may also have the opportunity to paint a picture of the ways in which they might play a role going forward in supporting the boss and getting their needs met, if they are interested in so doing. In all cases, you will want to consider fit, resources, and intention to help you select the best possible data-gathering tools.

Data Analysis

Sorting through feedback data is analogous to writing a research paper. Imagine you are writing a paper on Abraham Lincoln. As you read documents, visit museums, and conduct interviews with various historians, you are collecting a large amount of information. To make sense out of all the data, you begin to sort and organize the data into related topics—birth and youth, schooling and first jobs, political career, years as president, and death and legacy. Then, you keep sorting through your notes, editing and refining it into a coordinated and well-worded paper. The same process is necessary when analyzing data for feedback.

Once data have been collected, it is the coach’s job to sort the data so that clients can consider the information and see themselves as others do. It is important to keep an open mind as you collect, categorize, and link pieces of data. Taken separately, each piece of data reveals discrete information at given points in time. Collectively, data reveal themes, patterns, and inconsistencies that have to be worked through, analyzed, and presented to clients. It is critical to strike a balance of directness and care so that clients are energized, not paralyzed, by the information.

Let’s look at an example of raw data, in exhibit 5-1, collected from three sources about the subject, Craig, a project manager with a well-known technology company. He completed a self-assessment focusing on emotional intelligence. Craig, his boss, four peers, and five direct reports completed a 360-degree feedback instrument. His coach conducted personal interviews with Craig’s boss, three peers, and three direct reports.

It is difficult to read through all of these data points and distill them in such a way that encourages reflection and action. Let’s consider the information on Craig by breaking it up into pieces, starting with the self-assessment. Craig received a low average overall score on this emotional intelligence instrument. Based on this, what are two working hypotheses you might create? One hypothesis may be that Craig hasn’t had much coaching on or work to support his development in emotional intelligence. A second hypothesis may be that Craig doesn’t have an interest in emotional intelligence. The important consideration at this point is to keep your hypotheses fluid. You don’t know the whole story yet.

Now turn your attention to the 360-degree feedback. Consider the consistencies and inconsistencies in these data. What do you notice? Participative management is listed as “most important for success” by Craig’s boss and his other raters. Yet, Craig did not select that skill set as most important. Instead, Craig selected decisiveness as most important for success. If someone believes decisiveness is most important for success, how might he or she behave? What questions does this information raise for you? What are two questions you’d like to have answered?

Exhibit 5-1. Craig’s Data.

Finally, let’s review the comments that were shared during the image study interviews. What are some of Craig’s strengths? What do people admire about him? Where is he effective? What are some of the things getting in Craig’s way? How do people experience Craig? What contributes to this?

You might choose to break the comments into two lists, as shown in exhibit 5-2: what Craig does well and what Craig could do better.

Exhibit 5-2. Preliminary Sort of Craig’s Data.

So, why not just give Craig one big list of the things he seems to do well and the things he could do better?

It is easier to process or absorb information when we can organize it in some way. As David Rock points out in Quiet Leadership, “the brain is a connection machine” (2006, 3). The brain prefers to take data and connect them to pre-existing maps. These maps provide context and allow us to reflect on the data as part of a bigger picture. In Craig’s case, extracting what he does well and what he could do better might provide a workable basis for a conversation. Identifying themes in the data and sorting the data further to include hypotheses may create a richer conversation. Sorted data are easier to process and reduce the risk of overload. Whatever sort you use, you may choose to provide all of the verbatim comments, none of the verbatim comments, or selected verbatim comments to give depth to the themes you identify. Whatever you decide, it is important to be clear with your clients and the people providing feedback so that everyone involved knows how the information is going to be shared.

Another benefit of sorting data thematically is that the tendency to rebut or argue each point is substantially reduced. Themes are more powerful than singular examples. And, sorting the data increases the chances that Craig will be able to take action in specific areas. A second sort of Craig’s data could look like exhibit 5-3.

What differences do you note between these two approaches to sorting Craig’s data? How will these differences affect the feedback conversation you plan to have with Craig? The first data sort gives you two large topics to explore: what Craig does well and what he could do better. This might support a substantive conversation. The risk in using two such broad categories is that Craig may not be able to identify actionable areas. Or, if he does, it may be difficult to prioritize when so many are presented. Finally, the amount of data contained in the two categories is substantial. It is likely to cause overload and, in that case, Craig will be paralyzed, not energized to act on the feedback.

The second sort breaks the same data into more manageable chunks. It also presents the data with hypotheses, which Craig can accept, reject, or modify to help inform his thinking. Once the data are in pieces, Craig can fit and refit the information, getting progressively more comfortable with it. With his coach’s help, he can note patterns, themes, and make connections, ultimately deciding on priorities and actions he will take.

During the feedback conversation you have with Craig, you will want to explore the relationship between Craig’s behaviors and the results he is experiencing. Asking specific questions will help Craig focus on that link:

• How are people reacting to him?
• What behaviors are supporting him?
• What behaviors are getting in the way?
• Why did he decide to approach various situations the way he did?
• What led to the choices he made?
• What results does he want to experience?
• What has to change for him to get the results he desires?

As he responds to these and other questions, it is likely that Craig will reveal his thinking and the beliefs that led him to behave in certain ways, which in turn led to the results he is experiencing. Once he shares the beliefs that led to his behaviors, you can help him consider alternative beliefs and play out what behaviors would support them. The emphasis in this conversation should be future-focused. The work of analyzing deeply rooted motivations and pathologies belongs in therapy, not coaching. It is important though in your role as coach that you support your client in connecting the dots between his or her thinking, behaviors, and results. In doing so, you support your client in developing critical thinking and the skills needed to analyze the link among beliefs, behaviors, and results and to make changes in them as needed.

Exhibit 5-3. Secondary Sort of Craig’s Data.

As you work with your client’s data, allow your thinking to diverge as you consider alternative perspectives and rationales. Continue to ask yourself questions as you frame and reframe the data you’ve collected. Challenge yourself to identify links and relationships between different data points. Construct and deconstruct the pictures of your clients so that when you review the data with them, you will be prepared to challenge their thinking in the same way.

This chapter describes three primary sources of data, reasons to collect data, and some methods and tools used to collect data, and it examines the purpose of analyzing and sorting data, extracting themes, and constructing hypotheses. In addition, the chapter examines the link between behaviors and results, which will be revisited in chapter 6 when we explore feeding back coaching data.

Moving Ideas to Action

You can use the information presented in this chapter to increase your choices for collecting and analyzing data. Incorporating additional methods and tools will help you to provide richer data for your clients and support them in making informed choices. Use the template in table 5-2 to identify specific things you will put into practice.

Table 5-2. Data Collection Questions.