Asking the right questions about data

What you’ll learn

This short chapter gives you a set of questions to ask data specialists when discussing analytics results. It will help you structure the questions and answers part after a data scientist has presented his or her data to the audience. We will show you three kinds of questions to ask and how to ask them in the most constructive way possible.

Slowly but surely analytics practices are taking hold in organisations and most professionals are becoming familiar with data-driven decision making. They are using customer data to better target their sales, operational data to streamline processes, HR data to finetune trainings, or sentiment analysis to understand their social media impact, to name but a few examples.

But to make such analytics efforts successful, high-quality dialogues between business professionals and their analytics colleagues are crucial.

In our experience, asking the right questions about the data and its analysis is a key task to get the maximum value out of data and analytics applications (Figure 7.1). To support teams in their collaboration, we have thus compiled particularly useful questions that business people should ask their data scientists whenever they are presented with new data or discuss analysis results together.

The questions that we found particularly helpful can be grouped into three areas.

1. 1. Questions related to data sources and data quality.
2. 2. Questions relating to the data analysis.
3. 3. Application-related questions.

×

The first kind of questions are instrumental to better assess the validity and reliability of the data. The second set is helpful to understand what the data scientists have actually done with the data (and why). This prepares the ground for the third type of questions: application-oriented questions are probably the most important ones to turn insights into impact. They enable you to put the analysis results into practice. Still, you can only assess the answers to these questions if you have first asked the other two types of questions.

What are the benefits of asking these three kinds of questions? Let us just mention the main advantages of adding these questions to your analytics meeting repertoire.

• ●They help analysts and non-experts build common ground and avoid misunderstandings.
• ●They improve the rigour and relevance of the data analysis, by providing focus, uncovering weak spots, and subsequently improving the data gathering and analysis process.
• ●They assist you and your team in finding innovative new ways to harness your data.

So how can you reap these benefits? Let’s examine the repertoire of questions first, and then briefly describe the best way to use them. Read the list below as a menu to choose from. You will never have time to go through all of them in a meeting, nor is that even necessary. But in our experience, picking one to two questions per type makes sense in most data discussion forums.

For every group, we have put the questions in a sequence that makes sense and that allows the analyst to gradually open up (and not shut down or become defensive).

1. Questions related to data sources and data quality

Any analysis can only be as good as the underlying data. You must thus understand where the data came from and if it is fit for use. To assess the data sources and the quality of the data, you can ask your analytics colleagues the following five questions.

1. 1. Why are you focusing on this data? What questions does it answer and why are these questions crucial for us? In other words: What’s the value of this data?
2. 2. Can you tell me how, where (sources and their reliability) and when (time period) we have gathered this data? What biases, events, assumptions or preferences may have affected this data gathering?
3. 3. What do we know about the data’s reliability, consistency, completeness and accuracy? Is the data trustworthy in your view and current enough?
4. 4. What data are we missing that you wish we should have? Are there ways to get it?
5. 5. How much is the gathering of this data costing us? Are there ways to make this cheaper, more automised (for example through AI)?

2. Questions related to the data analysis

Many data analysts love to talk about their techniques, tools and how they generally conduct their work. It is thus important to focus the discussion on the analysis parts that really matter for the subsequent use of the data. The following questions can help you to steer the conversation towards useful data analysis aspects.

1. 1. What are the key terms and concepts that I really need to understand to make sense of this data? Please explain them to me as if I knew nothing about statistics.
2. 2. What were the analytical tools, procedures and assumptions that you have applied to this data and why those and not others? Anything I should know about these procedures?
3. 3. What surprised you when you did the analysis of this data?
4. 4. What was the greatest difficulty in analysing this data?
5. 5. What’s the finding you’re most sure about, and where do you have less certainty?

3. Application-related questions

Data should be a catalyst for decisions and actions. To help analysts turn data to decisions, ask them the following questions.

1. 1. What’s the key finding of your data analysis and why? What would you do in light of this data?
2. 2. What results could be easily misinterpreted or misused? How so? Where should we be careful regarding the generalisability of our interpretations?
3. 3. Who else should hear about these insights? What should they do with them?
4. 4. Is there any other way we could exploit this data and get value out of it?
5. 5. If we could start this data analysis over, what would you do differently now (to avoid errors, reduce cost, or answer additional questions or enable better application)?

A word of caution here: not all analysts see it as their role to make suggestions based on their data. Some see their job as mere data delivery and synthesis. This last set of questions should thus be asked carefully, iteratively and (most importantly) constructively (signalling respect and a willingness to improve together). Try to establish a collegiate atmosphere when talking about the data’s uses. This brings us to the final section on how to ask analytics questions.

Asking questions constructively

Having outlined which questions you should ask in analytics meetings, let’s now turn to how to best ask these questions. You certainly do not want your analytics staff (who may not always enjoy the communication side of their job) to become scared, defensive, or worried every time they meet you to discuss data insights. So, finding the right tone, time, and tenacity for your questions is imperative.

Your tone should be respectful, curious, and non-accusatory. The very advantage of the question format is that you are simply showing interest and want to know more, without casting a judgement or accusing somebody of not doing their job properly. Hence use the power of the (open) question format and avoid leading questions.

The timing of your questions should follow the process outlined above. So, start with simple, fact-based questions and gradually move to more complex and opinion-based ones. Also make sure that you time your questions well in the sense that you do not interrupt your analysts when they are presenting an issue that is clearly very important to them.

Regarding tenacity you should certainly challenge your analysts when they give you an evasive answer. At the same time, you should also show that you trust and respect them, for example when they have repeatedly given you the same kind of answer to a question.

Questions are of course not the only tool to use in such situations. As important as probing for data sources, analysis aspects, and decision consequences is acknowledging the work that has been done. So, don’t forget to give positive feedback to your analytics staff (and listen to their feedback to you) and thank them for their efforts. Frame data discussions as joint learning events and track how you can improve them continuously. In this way you will avoid the most common source of mistakes in management, according to its most revered guru, Peter Drucker, who said:

‘The most common source of mistakes in management decisions is the emphasis on finding the right answer rather than the right question’.

Key take-aways

Whenever there is a presentation of analytics or data among a group of people, make sure you reserve time for questions and answers. Pay special attention to these elements:

• ●Make the presenter feel comfortable and start with reassuring observations and appreciative questions.
• ●Make sure everyone has the same understanding. Ask questions that others might feel shy to ask.
• ●Start with clarification and fundamental questions before you enter detailed discussions.
• ●It is legitimate to ask questions about the reliability of the data sources and the trustworthiness of the data.
• ●Make sure you ask questions about the hidden assumptions that guided the data scientist’s analysis work.
• ●Probe the analyst about the action implications of the data and the interpretation limits.
• ●Give constructive feedback that allows the data scientists to improve their data presentation next time around.

Traps

Possible risks in data Q&A sessions are:

• ●Accusing the analysts of not doing their job properly, instead of helping them to improve their data delivery to decision makers.
• ●Creating an atmosphere where no one is willing to admit they did not understand the data.
• ●Having a pseudo or superficial understanding of presented data and asking only alibi questions (as Jennifer did in the opening dialogue).
• ●Offending or scaring data analysts so that they will no longer want to present in front of that audience.
• ●Asking leading questions that force the data scientist in a certain direction that the data itself may not warrant.

Further resources

Here are two great HBR articles on the power of asking questions:

https://hbr.org/2018/05/the-surprising-power-of-questions

https://hbr.org/2015/03/relearning-the-art-of-asking-questions

Here’s another take on the questions to ask your analytics team:

https://knowledge.insead.edu/blog/insead-blog/are-you-asking-the-right-questions-of-your-data-team-17056