CHAPTER
6
Interviews
Honing your interviewing process will allow you to build a competitive advantage. When you can differentiate well, you will identify strong, highly capable candidates whom your competitors will overlook or mismeasure.
Once you enter the interview stage, the total cost of evaluating a candidate increases substantially, so it’s critical to do it once per candidate and get it right. This chapter addresses measurement, building and running interview teams, scheduling and running interviews, phone screens, and on-site interviews.
All measurements from instruments in the real world have a margin of error, whether the instrument is figuring the distance between galaxies or the rate of nuclear decay in atoms. The measured system’s complexity and the desired accuracy drive the cost of producing a measurement.
For example, if you want to know the width of a table to within one centimeter, you can cheaply use a tape measure. If you want to know the width to within one micron, you need sensitive calipers and a temperature- and pressure-controlled environment. If you want to measure it within one nanometer, you’re going to need an electron microscope as well. Cost mounts rapidly.
People are a lot more complex than tables and have many more dimensions for potential measurement. Each dimension you want to measure adds cost, and each increase in accuracy adds cost. Without a bottomless checkbook, you must choose the dimensions and levels of accuracy you need and learn to work within a certain amount of fuzzy knowledge and some doubt.
You will not learn everything you want to know about candidates, but you can expect to learn what you need to know to make a sound hiring decision. It is important to be informed about the accuracy of your knowledge, and you should establish confidence by using the best instruments you can, which at this point are interview questions. You will still be in the dark about many aspects of each candidate, and worse, you cannot be totally certain what you don’t know about, as interview measurements will be inaccurate in unknown directions and to unknown degrees.
The simplest way to work with unknowns about candidates is to assume the worst and err on the side of hiring safety: When you don’t know, assume a candidate has weak skills, low capability, little knowledge, and a flaky personality. You will miss some good hires, but you will frequently avoid bad hires. The wider your margin of error and safety, the more candidates you will miss, particularly candidates with capabilities that are not obvious under examination.
Revisiting Candidate as Customer
When candidates are customers, the main feature of your service is the interview. That’s where you and they have the most interaction in the process, and it’s where the candidates’ participation counts most. The interview is not just an evaluation technique, it’s your best opportunity to establish your organization’s competence with candidates, build a reputation for a great experience, and lay the groundwork for candidates to view any offer you might present in the most favorable light.
Interviews are a focal point of your reputation because candidates tend to report about them to others. You must nail it, so put great attention on the candidate experience: Make the process as transparent as you can and as comfortable as possible considering the candidate is subjected to many difficult tests in succession.
Transparency with Candidate Guides
Customers want to know what to expect when they interact with your process. One way to set these expectations is to create and distribute a comprehensive guide to your interviewing process—a guide that reduces confusion and preempts questions that otherwise consume your time. Your guide may also address or counter stories and legends about your interviews in circulation. Such stories—the meat of your reputation—are usually reported by rejected candidates. They are often biased and negative, sometimes simply false, and likely out of date, so countering them with an inside perspective and facts gives you another means of building a good reputation as an employer that won’t frustrate and humiliate candidates.
I recommend you include any service level agreements you have committed to, such as how much time may lapse between an interview and a hiring decision. It further sets expectations and helps you stick to the commitment (just as when you’ve set expectations with a customer).
A recruiter once asked me for help preparing candidates. When I sent a comprehensive, multipage guide, she said, “This is the most complete answer I’ve ever gotten to any question!” That’s exactly the sort of relationship you want with recruiting allies: They know you’re totally and uncommonly committed to success.
There is an example of a candidate guide in the appendix. It details a specific process I’ve used and is meant to be a useful starting point. Write your own to reflect your experience and procedures. I send mine by email to candidates early in the process.
Candidate Experience Horror Story
This is a personal story of a poor experience, which shaped my attitude about customer experiences. You should be concerned if you recognize something of your own process in this.
In 2005 I had a series of phone screens with a large online retailer for a software development engineer position. They arranged three screens, one at a time, over a period of two weeks, and each had significant problems.
The first interviewer did not call. A few days later, it was rescheduled, and a rather haughty interviewer gave me an otherwise decent interview ending with a difficult brainteaser. This was deeply frustrating because the job is not about solving brainteasers quickly, and it’s easy to stop making progress on such a puzzle until you have a sudden breakthrough. If you don’t get it in the time allotted, you feel like an idiot.
The second interviewer called twenty minutes late and asked me some tricky technical questions that I definitely nailed, and then gave me a series of brainteasers of increasing difficulty. Overall, this was as frustrating as the first caller.
The third interviewer examined me for a system administration job, which is not my profession, asking a series of questions that I could answer only vaguely and refusing to respond to my concern that he was not talking to the right person.
All of this was followed by a no-thank-you call from a different impolite recruiter. The total experience was excruciating, so for the next three years I advised everyone I worked with or met who mentioned this particular company that those folks were a bunch of clowns who didn’t know what they were doing and not to bother with them. I don’t know how much damage this did to their recruiting pipeline (probably not very much). However, it’s often appropriate to assume your experience is typical, and this is borne out by many anecdotes about this company from that period, so they likely typically irritated candidates. That fact most certainly damaged their recruiting efforts for a long time.
Unless you’re going it alone—and you shouldn’t be—interviewing is a team effort that requires coordination. An effective team has multiple distinct roles, though as usual some may be filled by the same person wearing several hats.
Hiring managers should construct interview teams from trained and qualified interviewers. This section provides a description of the roles in an interview team, how to qualify and disqualify interviewers, training, and organizing interview teams in pools and for particular interviews.
Roles
Just as there are separate roles in the overall hiring process, there are definite roles in the interview. If you are explicit about who owns a role, you know who’s responsible for doing it well and you can reassign or delegate it all at once.
The coordinator makes sure all the other roles are in place, orders food and drink if necessary, and monitors the interview to verify that it is running smoothly. This person usually does simple setup, such as stocking a whiteboard with markers and an eraser. The role is sometimes filled by the recruiting process scheduler.
This person greets the candidate and starts the interview process for the day, either proceeding to an interview themselves, or escorting the candidate to an interviewer. If you need a nondisclosure (NDA) or other confidentiality agreement signed, this is the person to present and collect it.
Always choose interviewers who are roughly qualified in relation to each candidate. For interviewing software developers, this is almost always another software developer, so interview teams should be heavily stocked with working software developers and not overloaded with human resources or development managers.
You can help candidates find a comfort zone during interviews by using interviewers who are in some sense like the candidate. That may also make it easier for candidates to fairly evaluate offers, if they can see that the new environment won’t be composed of entirely alien people. Unfortunately, the effect works in reverse as well—interviewers may develop an unjustifiable positive bias toward the candidate.
In their 1999 paper “Unskilled and Unaware of It: How Difficulties in Recognizing One’s Own Incompetence Leads to Inflated Self-Assessments,”1 David Dunning and Justin Kruger described how people who have little competence in a given field frequently cannot recognize substantial competence.
A straightforward extension of this phenomenon (which I have witnessed many times) is that even otherwise competent people may not recognize superior competence. A junior engineer may not be able to tell the difference between a senior and a principal engineer and may even make the mistake of not recognizing competence at all. The approach and solutions created by a greatly superior engineer may have unrecognizable nuance and subtlety, anticipating problems the more junior interviewer hasn’t seen and couldn’t foresee without guidance.
My interpretation of the Dunning-Kruger effect and my experience are that with a little training, most people can fairly evaluate candidates who are a little bit more or a little bit less skilled and experienced than they are. Evaluating engineers outside this natural range takes practice, and not everyone seems to develop a knack for it.
Some interviewers will be particularly good at interviewing certain kinds of candidates and not others. For example, I know a principal engineer who excels at evaluating college candidates but consistently gives overly critical evaluations of industry veterans.
The language of the interview is also an important consideration. The mental gymnastics of an interview are complex enough without forcing candidates to speak in languages they cannot use fluently, so use interviewers who are fluent with the language the candidate is expected to use in the ordinary course of work (such as English).
Often overlooked, there is a special challenge to hiring for new roles and specialist roles—the sort that you don’t have on your team already. That might be someone like a molecular biologist to design computational models for a new gene network analysis tool or just your first software developer. For candidates with fields of expertise you have difficulty evaluating, you may be able to find and ask questions that match the skills you need for the role, but interpreting the answers is another issue. Careful interpretation generally requires another specialist.
You can make do with what you have if the new field of expertise is relatively close to ones you already have. It may be more effective if you can provide some training, such as teaching software engineers to evaluate software test engineers.
If you still have no qualified interviewers, find someone you trust who is qualified to evaluate the candidate in the area of specialization required. It could be someone from another department, a consultant, or someone you know personally. If none of these are available, you might substitute qualification signals, such as peer-reviewed publications, followed by candidate references. Finally, you may have to do your best to determine whether candidates are at least comfortable and articulate when discussing their specialty and make a hire on faith, then evaluate the success of the hire by his or her performance.
Another option is to take on a specialist as a contractor, if the person is willing. You are stuck with the same evaluation problem, but it may be cheaper to conduct trial-and-error searches.
Some people should not be allowed to interview candidates. You’ve met them: the disgruntled, the underperforming, and employees on their way out, as well as anyone who feels threatened by bringing on new employees. Highly introverted engineers and the chronically surly tend to be poor interviewers and make negative impressions on candidates.
There’s a pernicious type of employee who uses interviews as opportunities to show how much smarter they are than candidates. These egotists will ask questions that rely on obscure knowledge or that are inappropriately complex, or will hold answers to an impossibly high standard. They have no interest in actually evaluating candidates. They can be overbearing or intimidating. These interviewers are dangerous to your hiring process, so root them out and cut them out of the process as soon as you can.
Don’t be afraid to take troublesome people off interview loops and out of your interview team. If an interviewer has an irreconcilable difference of opinion with you over basic procedure or evaluation philosophy, you should certainly hear them out, but don’t let it become a source of continuous frustration. Make the cut and move on.
Some people become excellent at interviewing, developing superb intuition and critical insight into candidates with little training. Others have already received training before you met them and gained experience that honed their interviewing skills to a fine degree, so they will need little or no help from you. In either case, such people are rare, so it’s likely you will have to build this capability in interviewers.
All interviewers must understand the interviewing and hiring policies of the company and applicable laws, including the types of questions they must not ask, questions they should not answer, and unacceptable interviewing behaviors. They should know the essentials of the recruiting process and know to treat candidates as customers. Additionally, your team’s interviewers should share an understanding of the local process and the team’s standards. The more they know beyond that and the more skills they accumulate, the better they will differentiate between candidates. Various methods are available for making great interviewers.
In my experience, having new interviewers shadow interviews is the most effective way to build competence and confidence in their ability to effectively interview and evaluate candidates. It exposes them to the real deal in a relatively safe manner and gives them examples to draw from, and access to experienced interviewers to confer with and ask specific questions.
Shadowing a phone screen is simple. Put another person in the room with the interviewer and conduct the interview via speakerphone, with the shadow remaining quiet. It isn’t necessary to introduce the shadow. Both the interviewer and shadower have the opportunity to press the mute button for short periods to comment on or discuss the interview as it progresses, and the candidate will not need to be concerned about having a third party listen to the conversation.
The presence of a third person during an on-site interview can make candidates uneasy, but they usually relax if they understand the purpose. Introduce the shadow at the beginning of the interview; the shadow should greet the candidate and say something to the effect of “I’m here to sit quietly in the back and learn about interviewing. Please pay me no attention.” Then the shadow should do just that and take notes. You might also consider asking candidates before they arrive whether they would mind having shadow interviewers.
In either circumstance, the shadow and interviewer should discuss the interview immediately after, talk about the techniques and questions used in the interview, and share their evaluation of the candidate. When I am the primary interviewer, I try to ask the shadow for feedback on my performance. Feedback from fresh perspectives has helped me improve my interviewing in many ways, such as pronouncing words more clearly, speaking at a steadier pace, and patiently drawing more detailed answers from candidates.
Sending a new or experienced interviewer to shadow interviews conducted by another team may also be enlightening, showing new techniques and ideas. Your interviewer may be able to provide useful feedback in return.
How much you can teach via distributed documents, lectures, and hands-on workshops will depend on your organization’s culture and your team’s learning styles. My experience is that hands-on workshops are an effective way to distribute important structured information and build skills at the same time.
It’s almost certainly a worthwhile investment to conduct a small workshop for your team: reviewing the fundamental theory of hiring and highlights of how to hire, modeling questions and evaluations, practicing interviewing each other, and giving an opportunity to ask questions in context.
In some organizations, nothing is real until it’s written down. Whether or not that is your situation, it is true that documents are longer-lived than oral instruction, workshops, examples, and word of mouth. Writing down the essentials of your process in a friendly, easy-to-follow guide will help create a consistent system of concepts that will cause effective hiring, allow people to teach and refresh each other, and create a wake behind you.
I suggest you document, distribute, and archive only what you think people on your team will actually read, which in most circumstances is not very much. Consider writing or gathering a summary of the interviewing process from first candidate contact to offer, a summary of the organization’s basic hiring philosophy and standards, a list of examples of good questions, a phone screen guide or sample transcript, and a set of advice on evaluating candidates. Make these things succinct to increase the likelihood that someone will use them.
I have also asked interviewers to shadow in reverse, sending a great interviewer to observe a less experienced interviewer. Then they discuss the interview. This coaching provides an avenue for constructive feedback and leads to rapid improvement, as long as the two interviewers have a strong relationship and both see the process as collaboration.
I have found that shadowing and coaching gives my team strong interviewing skills quickly, and practice interviews reinforce these skills. These are simply role-playing sessions in which one person takes the role of an interviewer and the other the role of a candidate. Most people have fun with this kind of practice.
The technique appears to be particularly useful when you limit the mock interview to asking and answering a single technical question. It helps the interviewer gain practice at asking a particular question before asking a candidate, and it helps calibrate the question to the team’s needs. (Chapter 7 has more information on question calibration.)
Tracking and Profiling Interviewers
On an interview team there will be interviewers with various different personalities, strengths, and weaknesses. Taking full advantage of all the interviewers will require understanding and tracking their differences—such as who usually gives overly negative feedback, who can interview SDETs, and so on. A spreadsheet may help you organize this information as well help you select and organize an interview team for any particular interview.
Standing and Dynamic Interview Teams
An interview team can be composed of the same people for interview after interview—a standing team—or it can be a temporary team, drawn together for one interview from a pool of interviewers. Both approaches have advantages: a standing team establishes a rhythm and may develop a more consistent evaluation style (not necessarily a better one); a dynamically composed team may spread the interview load more widely and in my observation develops a more creative approach to evaluation due to differing styles that are not yet comfortably meshed.
Scheduling a standing team is straightforward because you know who needs to attend. Putting together dynamic teams from a large pool of interviewers requires careful consideration for each interview. A meta-schedule can help guide building each individual interview from an interviewer pool.
A meta-schedule is a set of instructions for creating and scheduling a team out of many possible team members. It should be comprehensive enough that anyone with access to the team members’ calendars can set up the interview meetings. The instructions can have several line items, each of which says “draw x from y” where x is a person and y is the interviewer pool. The following example should make this clear.
For each interview, schedule four contiguous interview sessions at fifty minutes each. The order is not important, except that where it is convenient we would prefer to have the hiring manager interviewing first or last. Schedule all of them either before or after lunch, but if scheduling over lunchtime is necessary, schedule the hiring manager for the lunch session.
Sessions:
- Hiring manager (required) and one from Pool A
- Two from Pool B
- Two (others) from Pool B
- Two from Pool C and one shadow from Pool D
Interviewer pools:
- A: (technical program managers): Bujor, Christi, Devon (in order of preference)
- B: (SDEs): Eliza, Franklin, Genna, Hiram, Ilsa, Johann, Katrina (pick randomly)
- C: (Senior SDEs): Bujor, Ilsa, Larry, Melissandra (pick randomly)
- D: Namit, Ophelia, Patri (shadowers)
Interview Structure
Chapter 3 explores the overall structure of the recruiting process and how you can create, adapt, and tweak it. This section discusses the internal structure of interview elements of the process: prequalifying, phone screens, and on-site interviews.
Prequalifiers: Barriers to Entry
To focus their time on candidates more likely to pass interviews, some hiring managers place proof-of-competency tests at the head of the candidate pipeline. The approach is most appropriate when you receive a big ratio of unqualified to qualified solicited applicants. If the ratio is not huge, but you feel pressed for time, this might work for you.
Prequalifiers are usually assignments: tests that candidates take and submit along with their application before it is considered, or as a preliminary step before interviews. They can be questionnaires, multiple-choice tests, coding assignments, design exercises, and so on. These barriers reduce the number of applicants, possibly quite dramatically. The bigger and more complex the assignment, the higher the barrier, so fewer engineers will apply.
In this sense the theory is sound: Engineers who cannot complete the assignment will not become candidates. It may also keep out engineers who are qualified but are simply too busy to take the assignment.
From the candidate’s perspective, she has many jobs she can apply for. A barrier that requires substantial effort to cross may keep out the engineers with the most choices; these are often the best engineers.
The developers who get through the prequalifying barriers are the ones who have an overwhelming interest in your particular job, as well as those with enough time on their hands. Having a lot of time on one’s hands isn’t always a good sign in a candidate.
The purpose of a technical phone screen is to admit candidates who have a certain chance of passing an on-site interview. You should set a target ratio and publish it to the team, such as one in four: one hire per four in-house interviews.
The ratio you aim for is your balance point for interviewing efficiency, determining how much time, on average, you spend evaluating each candidate. More on-site interviews per candidate costs you more interviewer time, so that’s the trade-off between avoiding interviewing people who are unlikely to pass your interviews and accidentally failing to continue interviewing great engineers.
Calling the candidate is easy, but conducting a useful phone screen takes planning and practice. Each role you interview for has idiosyncratic requirements, but in general for software engineers you should plan to learn:
- Whether the candidate can communicate clearly,
- Whether the candidate can code, and
- Whether the candidate can create and think about algorithms.
These capabilities are broadly—perhaps universally—useful, and understanding a candidate’s ability will let you roughly estimate the odds that he or she will pass an on-site interview. To learn about these capabilities, develop and include questions that you find over time will differentiate between candidates who pass on-site interviews and those that do not.
To keep the admittance to on-site ratio at your target, use feedback from further along in the interview process. Do you typically end up rejecting engineers after the on-site interview because they cannot code well? Step up the difficulty of coding questions on your phone screens—or at least reevaluate the ones you use. In the other direction, to increase the number of candidates who get to on-site interviews, you can relax difficulty in areas that are not common failure modes for candidates you interview.
Using a consistent structure for phone screens will make it easy on you and the candidate, covering all the important points and leaving your attention on the important part: learning what’s different about this person. There’s a full sample phone screen transcript included in the appendix. Here is a highly compressed abstract.
1. Greeting
- “Hi, this is Patrick calling from ExampleCompany. May I speak with Chaz Fernandez? . . . Is this still a good time to talk?”
2. Introduction and Verification
- “I’m a software development manager and I’m calling to do a phone screen for a software development engineer position. Is that what you expected?”
- “Do you have something to write with or an Internet-connected computer?”
3. Set Expectations
- “I’m going to introduce our company and team, ask you some technical and nontechnical questions, then let you ask me whatever you like.”
4. Brief Overview
- “ExampleCompany is in the software as a service industry, and our team is responsible for developing and maintaining a public distributed computation and caching service.”
5. Interview Questions
- “Please write me some code that . . .”
6. Candidate Questions
- “What would you like to know about us?”
7. Next Steps
- “I’m going to review my notes carefully, and we’ll get back to you shortly.”
8. Wrap-Up
- “Thanks for your time today. Have a great evening.”
9. Prepare Feedback or Record Notes
I recommend keeping notes as you conduct a phone screen, which you can refer to later when you prepare specific feedback, update a candidate book, or prepare for a hiring decision. Typing them directly into a computer saves some transcription time, and I use a template to do so. My template is an otherwise empty document with a list of the questions I intend to ask. Notes go in between; I save the document, and I have everything I need to create feedback and pass the notes on to future interviewers.
I have also found myself wondering after an on-site interview, “Why did I decide to bring this person in?” Thorough interview notes provide a clear (if not always flattering) answer.
Phone screens can be strange and scary even to experienced interviewers, and it seems that both sides of the conversation are at a disadvantage. First, you talk with a stranger. Second, there’s a substantial power imbalance between the participants. Third, the participants have differing motives. Fourth, the communication channel is unnaturally narrow.
Normal human interaction is substantially nonverbal. Stance or posture, direction of attention, placement and movement of hands, eye contact, facial expressions, micro expressions, and much more set the context of communication and reframe it as the conversation goes on. All of that information is stripped away when we talk on the phone, and we are left with only auditory communication cues such as inflection, timing, volume, and words.
Because nonverbal signals guide others to interpreting our words as we mean them, miscommunication is much more likely over the phone. You may have noticed this. Be careful to speak clearly and listen generously.
Solving problems over the phone seems to be universally more difficult than solving them in person, so an otherwise simple question can become difficult, and a tricky question can become virtually unsolvable. Some classes of question are nearly useless in a phone screen; coding problems in particular require special care.
In person, a candidate can closely monitor the interviewer for clues and feedback as they start to respond, asking clarifying questions or producing an answer. Over the phone, uncertainty does not resolve easily, so candidates frequently go off track when presented with slightly vague or ambiguous questions. When they are on track, they will often wonder how they are doing.
I find that the interview becomes a lot more difficult when asking questions that involve large design spaces, ask the candidate to work in an unfamiliar domain, or can only be answered well after several rounds of back-and-forth discussion or extensive requirement gathering by the candidate.
Great phone screen questions are briefly posed and briefly solved, easily stated, and have an answer form that’s either highly verbal or expressed in short pieces of code.
Knowing what not to ask about will save time and keep unimportant information out of your head. I don’t ask:
- All about their most recent project.
- A litany of skills and buzzwords.
- What they did on their summer vacation.
- Their childhood pet’s middle name.
The answers to these kind of questions are distracting, take valuable screening time, and are usually not relevant. If you really need to know any of that stuff, you can cover it on-site.
A major class of questions to avoid asking in phone screens is any open-ended inquiry about the candidate’s experience. Such questions pass control of the conversation to the interviewee, and you really shouldn’t let that happen. You are in charge, and the candidate should spend their time answering your questions or asking their own at your invitation, not retelling stories. When the candidate controls the interview, the time will be spent on their interests, so you won’t learn what you need to know.
On the phone it’s critical to ask coding problems that can be easily and readily explained, coded quickly, and evaluated quickly. I only ask coding questions that can be posed in less than one minute (usually less than thirty seconds), which good candidates can understand rapidly with no confusion. These questions have elegant solutions that are no more than a handful of lines of code, giving the candidate a fair chance to design and code a solution. Wide varieties of coding problems meet these conditions and differentiate among candidates, highlighting those most likely to succeed.
Chapter 7 has more discussion of designing, choosing, and evaluating interview questions.
My personal practice is to keep coding questions short and straightforward, such that most solutions can be written with a few lines of code. These can be worked out easily with pencil and paper.
Candidates can acquire and use simple writing tools wherever they are. They are also kept off computers; where there may be a temptation (or even an involuntary reflex) to look up answers and other information they may use to perform better on the question than a candidate without these resources.
The disadvantage is unfamiliarity; it is uncommon for software developers to work out code solutions in anything other than an IDE or at least a text editor. It may feel a bit unnatural for them. Acknowledging up front that it’s going to feel unnatural relaxes most candidates.
Screen-sharing lets you share screens or live online documents with a candidate. This gives you the opportunity to see her work progress as well as the end product immediately, with much less chance for miscommunication. No one has to read code out loud over the phone. It is especially useful for code that is verbose or complex, such as regular expressions or HTML.
However, a downside is that shared screens require the candidate to have an Internet-connected computer with them, sometimes particular software installed, and a hands-free phone or headset. That limits the circumstances under which they can screen. The computer in front of the candidate also gives her an excellent opportunity to quickly look up information to respond to questions you otherwise expected them to answer without aid and tempts many to cheat by searching for source code.
Personally, I’ve never been sure of what I have learned by watching candidates type. Everyone seems to think a slightly different way, often nonlinearly, and this makes them jump back and forth, refactor, rename, and so on. All I can learn is that they seem to be actually working on it, and I see little else of use about their process and results that can’t be learned in other ways.
There are many ways to accomplish screen sharing using tools such as Google Docs (http://docs.google.com), See[Mike]Code (http://i.seemikecode.com/), and general desktop sharing applications such as Fog Creek Copilot (www.copilot.com).
Some difficulties come up repeatedly with phone screens. These are problems I have coached people through many times: staying on too long, working with senior candidates, handling candidate expectations, and candidates who talk too much.
You wisely arrange your phone screens to cover the truly important and most differentiating aspects of the interview first, so you may have the opportunity to detect early that a candidate simply will not pass the rest of the phone screen. You need to get off the phone because you have other things to do— like find more candidates.
Cutting a phone screen short can feel quite rude, and if you do it abruptly and with no compassion it will be a poor experience for the candidate. Candidates seem to universally appreciate honesty, and their time is valuable as well. When you are certain there’s no hope, go ahead and let the candidate know. Avoid platitudes or false flattery. For example, here is how I halted a recent phone screen:
“Based on how the interview is going so far, I think it’s pretty clear that this position requires substantially more fluency with algorithm design. So there’s not really a match possible here. I know your time is valuable, so what I’m going to do is end the interview now. Thank you for talking with us.”
Senior Candidates and Complex Questions
Experienced engineers accumulate lots of information and many skills, which is why you want to hire them, and they often expect to have these skills put to test in interviews. Phone interviews, however, are not a very good place to ask the complex and nuanced questions that will engage a candidate’s deeper knowledge and hard-earned skills, so they may feel underchallenged. It is not rare to hear a senior candidate report that they felt they were interviewed for a junior position rather than one that would challenge them and demand their skills.
I simply advise them that the purpose of the phone interview is to screen out candidates who are unlikely to pass an on-site interview, and it’s the basics that commonly drive on-site interview failures. At the on-site, we ask the sort of questions that will demand and exercise their well-developed skills.
Candidates may have preset notions about how interviews should function and what their role is. For example, they may expect to be asked detailed questions about past projects, and if your interview does not meet those expectations, they may come away disappointed or frustrated.
You can only spend so much time setting expectations, so you will inevitably disappoint some people. A candidate guide, as described earlier, can help inform people about your intentions, though this won’t stop them from disagreeing with your methods if they choose to do so.
Candidates Who Have Something to Tell You
I don’t ask for it, but if they volunteer, I let candidates spend a few moments telling me why they are a great fit for the job. However, I politely stop them before they get very far. It’s not a question I asked, and the answer is usually not enlightening. I know from their résumé what they have worked on recently, and they demonstrate the skills and capabilities that are the most important for the role by performing during the interview.
Once you decide to bring someone in for an on-site interview, costs mount quickly. Your organization invests the time of all interviewers for the interviews, their feedback preparation, and recap meetings, as well as whatever capital it takes to bring candidates to your location. With this much at stake, you want to get it right and do it the minimum number of times per hire.
Organizations have various approaches to locating interviews at their sites—from individual offices to cafeterias to conference rooms.
Individual offices have the merit of being easily booked, because they are usually plentiful and interviewers often have their own. In their own offices, however, interviewers may be “too comfortable” and too easily distracted. Email and visitors are constant distractors, and it is very natural to have the candidate sit across the desk from the interviewer. Desks create a psychological barrier that may hinder effective communication,2 so when I must have a desk or small table in a room I try not to sit across from the candidate, but side by side or at a corner.
Cafeterias are common settings for interviews, particularly during lunch. But cafeterias are designed for getting a large number of people fed quickly, not for privacy or quiet. In a crowd, people tend to raise their voices to be heard, and raised voices can easily lead to raised emotions. I recommend you bring lunch to a candidate in a more intimate setting, such as a conference room or office, or at a quiet restaurant where you can talk naturally and the chairs are not designed to make people uncomfortable after twenty-five minutes.
All in all, I strongly recommend using conference rooms.
Schedule a conference room or spacious office for the duration of the interview, with lots of natural light and comfortable furniture. Using one room for the entire interview day is efficient and reduces the chance of scheduling errors, getting lost, or spending valuable time navigating between rooms. The candidate will appreciate not being yanked from room to room and will have the opportunity to get more comfortable as the day proceeds.
Ensure that you have lots of whiteboard space—whole walls of whiteboard, if possible. If you find yourself with a tiny whiteboard, you can make do, but you’re better off bringing in a large portable board from somewhere else or, if you have time, nailing a bigger board to the wall.
Evaluate your interviewing facilities for general accessibility and in particular for any candidate you know has a specific disability. Not everyone can stand at a whiteboard for long periods of time, or indeed stand at all. Not everyone can hold a marker steady, or hear you clearly, or hear you at all. And so on with any number of disabilities you may have to accommodate.
I was recently embarrassed to realize that a whiteboard I found uncomfortably high was impossibly high for one candidate, who could barely reach it, and yet it was a whiteboard-intensive engineering interview. Afterward, the facilities team brought in a portable whiteboard that went to the floor, but it was too late to avoid making a poor impression on the candidate as well as complicating our evaluation.
Disabilities are not always obvious or volunteered. On one occasion I had a team full of people who disliked a candidate because he spoke very loudly during interviews; interviewers felt that he was trying to be intimidating or had underdeveloped interpersonal skills. Someone realized that he was probably losing his hearing. He had asked all of us to speak up at one point or another—so we were looking at a disability, not a distasteful personality trait.
Human endurance is alarmingly finite, so you should organize the interview to get the most information you reasonably can about the candidate without spoiling the data with overexhaustion. Four to six hours with several breaks seems to be the most you can expect. Even so, you may find the candidate flagging near the end of the interviews.
Candidates who have to travel far to the interview site may start with fatigue or jet lag, so factor this into the length of the day and your start time. If you have to fly candidates quite far, it’s a good idea to book an overnight stay and an interview the following morning.
Number and Length of Interview Sessions
Introductions, establishing a rapport, and creating interview rhythm takes several minutes of each session. Set aside a few minutes for candidates to ask questions in each session, and expect to spend some time each hour on breaks and fetching beverages. If the interview sessions are not all in one location, travel also consumes time.
Answering technical questions requires substantial concentration, so it is difficult to get a technical read on a candidate with a session length of less than forty-five minutes. More than sixty minutes seems to fatigue candidates and interviewers, so a session length of forty-five to sixty minutes works well most of the time.
You may be able to use shorter spans for interviews focused on nontechnical matters that don’t require intense or uninterrupted thinking time. This might include interviews that focus on collaboration or for sessions with HR or managers who look for background information, ask about compensation expectations, or intend to answer general questions. Generally, it seems that thirty minutes works well for nontechnical sessions.
How you transition from one interview session to the next seems to have a surprisingly large effect on the process of evaluation. When an interview session is finished and the candidate is ready for another round, interviewers have several options for passing on what they have learned to the next interviewer. They can go without comment, provide preliminary feedback to the next interviewers or the hiring manager, or broadcast their feedback to all remaining interviewers.
Each approach has merit, and what you do may come down to a matter of organizational style and history.
The silent hand-off lets each interviewer form an unbiased opinion (at least unbiased by feedback from other interviewers).
Hand-off with comments gives interviewers an opportunity to pass on suggestions as to what topics they may want to explore, perhaps because their interview did not cover them well, or to look for even more depth in an area of strength. It has the drawback of being preliminary. Passing comments along can also cause a bias cascade, in which one interviewer causes the next to start the interview with a bias. That prior bias affects the interview, which affects what the interviewer tells the next one, and so on. The first interviewer of the day can have enormous, nearly invisible influence with this effect.
Broadcasting comments can give all future interviewers for the day the same information. It may not reach them all before the interviews, but it is particularly effective at spreading bias.
The silent hand-off model is currently prevalent for hiring at Amazon.com, and both the comments and broadcast models are commonly used at Microsoft.
Choose a method or hybridize to suit your purpose. On my interview loops, I ask the interviewers to contact me if they strongly believe that a candidate will not succeed, so we have the opportunity to halt early if appropriate, but otherwise not to pass on comments throughout the day, or to discuss their opinion with other interviewers before they have settled on their own decisions and written feedback.
Interviewers that hear about candidates before their own interviews are automatically and unavoidably biased by the people they talk with. Silent hand-offs are an excellent way to eliminate this bias.
It’s easy to interview candidates and have lunch at the same time, but it’s also easy to misuse the time. The candidate and interviewer are distracted by their meals, so it shouldn’t be used as a regular interview session. Don’t ask critical technical questions at lunch. Don’t ask coding questions. Let the candidate eat; take turns asking nontechnical questions so you can both eat. Use it as a selling time.
Feed the candidate. Don’t make the candidate pay for lunch or anything else during the evaluation process. Making the candidate pay will make you look cheap, thoughtless, broke, or all of the above.
Every interviewer in the loop should know what role you are trying to fill on your team and your hiring bar. They should have a copy of the job description (or whatever variant of that concept you are using), the candidate’s résumé, notes or transcripts from phone screens or previous on-site interviews, and the entire interview schedule.
You should also brief team members on your hiring philosophy as well as lay ground rules and expectations. For instance, you might remind the interviewers that you are not hiring for skill in a particular language but facility with any of several languages. You might tell them when you expect to receive written feedback following the interview.
This is a good time to make sure you’ve disqualified all the interviewers who will not perform the way you need, as described in the section “Disqualifying Interviewers.”
I’ve seen effective reinforcement of this approach when hiring managers send a morning-of-briefing email with the documentation and a reminder of who is responsible for what aspects of the interviews.
Maximize the use of interviewer time by avoiding unnecessary repetition. Repetition that leads to stronger analysis is fine, but repeatedly asking the same question or close variants of it can waste valuable interview time.
Set expectations explicitly by briefing each interviewer, letting them know what information you expect them to have when they leave the interview. A question plan, described in Chapter 7, will help prevent duplication and encourage covering all important candidate capabilities.
You need enough interviewers to evaluate every aspect of the candidate’s performance that you care about. In my experience, this is five to six people, including a development manager. Interviewers who have conducted more than fifty interviews can pull a lot of information from candidates, so it’s safe to use four to five experienced interviewers.
Fit the number of interviewers to the job requirements. If the role involves working with groups, such as an architect who must present designs and defend decisions to executive teams, use at least one committee interview with three or more interviewers. This forces the candidate to perform as they would on the job.
In general, committee interviews can create a layer of difficulty that may obscure the candidate’s capability. The candidate must not only consider and answer questions but also efficiently switch attention among a group of people. Without context about the interviewers’ social structure and the varying communication patterns and cues they use, this might result in hurt feelings (“He ignored me!”) or “social overload” for the candidate. There’s also the potential for a group/outsider effect occurring, in which the team unconsciously bands against the “interloping” candidate.
If you have inexperienced interviewers, or if you need to gather many different opinions but have limited time, use pair interviews. Two interviewers (not just one shadowing the other) at a time can be quite effective when they know each other, work together, and ideally have done pair interviews together before.
Most of the time I use single interviewers in each session, sometimes with a shadow interviewer attending.
Running the Interview
Great customer service is driven by attention to detail and empathizing with the candidate/customer.
- Have an NDA prepared and ready, in case of incidental discussion and observations, contents of whiteboards, and any comments about the future or anticipated directions, technologies, or products.
- Send a copy of the NDA to the candidate ahead of time so they have a chance to review it.
- Stock whiteboards with markers and erasers, and stock any other consumable or portable resources you’ll need.
- The coordinator arranges for a greeter to meet the candidate in the reception area, the lobby, parking garage, lounge, arrival gate, taxi stand, sidewalk, and so on.
- The coordinator arranges for a single interview room or, if necessary, two rooms, before and after lunch.
- Greet the candidate warmly.
- Confirm that it’s still a good day for the interview (no pressing concerns).
- Ask when they have to leave.
- Offer the candidate water, coffee, and whatever else you have available.
- Offer a trip to the restroom.
- Present and ask the candidate to sign the NDA.
- If appropriate, present a lunch menu or restaurant options. This will let the candidate take dietary restrictions into account without requiring them to volunteer that information, which you don’t otherwise need.
- Validate parking, and take care of other such logistical issues.
- Set the candidate’s expectations by saying how many interviews they will have (if there is a fixed number) and how long each interview will last if they are fixed in length.
Sessions
Each interviewer should prepare a list of questions that would normally greatly exceed the length of the interview. That keeps them from running out, which can occur for any of a number of reasons, such as the candidate admitting to having prepared for a particular question, discovering that another interviewer has already asked a similar question, or that one or more prepared questions turn out to be inappropriate for the candidate or circumstances.
Just as candidates are well advised to prepare for common or expected interview questions, interviewers are well advised to consider and prepare responses for common candidate questions. Interviewers should speak from the heart and be honest, but organizing thoughts ahead of time will make responses succinct and internally consistent.
Candidates frequently ask:
- Why do you enjoy working here?
- What is the work/life balance?
- What is a typical day like?
- What are the opportunities for advancement?
- What is your group working on? What’s next?
The last question deserves particular thought because interviewers should not divulge business plans to candidates no matter what confidentiality agreement they have signed. In my opinion, NDAs are really a formality, because once a secret is out, it’s out. Interviewers should give direct and honest answers that reveal nothing confidential.
During Session
The basic process is to make the candidate comfortable, ask challenging questions, and give them the opportunity to conclude that you are awesome. Here’s the basic structure of a typical interview session.
Greet: Give your name, describe your role and team.
- Offer break and beverage.
- a. “Can we get you some water or coffee or similar? Or a trip to the restroom?”
- Ask questions.
- b. “Create an algorithm for connecting DNA segments found via shotgun sequencing.”
- Answer candidate questions.
- c. “What would you like to know?”
- Sell the organization and job.
- d. “Let me tell you more about why you should work here . . .”
Hand off to next interviewer or escort candidate out.
Interviewers should keep a clock in view or, if no clock is available, check the time on a watch or phone. Don’t be discreet about it—announce it. In some cultures, discreetly checking the time is a social signal that you’re bored or have something else to do. Saying, “Excuse me, I have to check our schedule” preempts that signal.
Do not leave the candidate alone for more than a few moments, unless they are in a one-exit conference room and you’re on the other side of the door or you’re waiting for them outside the restroom. This accomplishes two important goals: to make the candidate feel welcome and cared for, and to minimize security risk. Candidates left alone can readily walk away with secrets or other valuables, whether or not they signed an NDA. Each interviewer has the responsibility for a live, in-person hand-off of the candidate to the next interviewer or to escort the candidate out.
A major purpose of intentionally designing and managing the interview process is to respect and optimize your team’s time spent on interviews. The amount of engineer-hours invested in an interview increases throughout the day, so it’s worthwhile to check whether you can save time and frustration for all parties (interviewers and candidate) by halting the interview day early.
- Have a manager interview before calling a halt.
- Protect against flukes by having more than one interviewer interview the candidate before making this call.
- Let only hiring managers and experienced interviewers decide whether to stop.
With enough interviews, you can safely assume that anything could happen: Candidates will be injured, the building will catch on fire, and so on. Be your normal, well-prepared, level-headed self and everything will be fine. When dealing with an unusual circumstance, keep in mind that the candidate is a customer and deserves deference, all else being equal.
Certain situations seem to consistently trouble or confound new interviewers. Here’s how I interpret and deal with some of the ones I encounter from time to time.
Candidate Acts Inappropriately
Check with HR on this point. In my jurisdiction, it is the employer’s responsibility to take reasonable action to stop workplace harassment,3 including when it’s caused by an outsider, such as a vendor or a candidate. Interviewers should feel free to cut their interview short if they are offended by candidate behavior.
Candidates that make off-color or otherwise inappropriate jokes in interviews will likely be even more informal after hiring. I never take a risk of that happening. Rude language, on the other hand, is frequently an automatic habit that seems to be easily retrained.
Your safety and your team’s safety are more important than any candidate or interview. If you or any interviewer feels threatened, halt the interview immediately. Call security (or police) and notify HR so they can record the incident and proceed with any necessary next steps.
Candidate Attempts to Take Over
Act and speak with confidence and you are likely to retain control of the interview. But sometimes candidates don’t understand, expect, or respond to the structure you try to create for the interview session. On a few occasions I’ve had to reestablish control explicitly by telling the candidate, “That’s fascinating and I wouldn’t normally interrupt you, but I do need to ask a few more specific questions due to limited time.”
Assertive personalities are fine and valuable, but there is a social context and business purpose for interviews that candidates must respect. When they don’t respect your guidance of the interview and work in the context you provide, they can interfere with your ability to evaluate them. More worryingly, it signals a personality that may not collaborate effectively.
Depending on how you define lying, 28 percent to 90 percent of undergraduate job candidates lie during interviews,4 and it seems a safe assumption that veterans might lie at a similar rate. When you discover a lie, the interview could go off the rails immediately and never get back on track.
I have encountered different philosophies for dealing with lying, from being blasé about it to being extremely critical. Most of the time I aim for more of a middle ground. (Some of this may be due to a direct personal experience I described in Chapter 5.)
When I spot an apparent lie, I take a moment to verify the facts and suss out assumptions. I ask a clarifying question to find the root, and then ask the candidate directly to reconcile the contradictory evidence. Most candidates make some sort of admission at that point, ascribing it to a misunderstanding, misstatement, or unintentional exaggeration. Whether this is really the source of the problem is a judgment call, but it gives candidates a fair opportunity to address the contradiction, and sometimes they clearly resolve the issue right away.
If the discrepancy is major and unresolved, I politely point out that the contradiction is worrisome and has raised serious doubts about the candidate’s trustworthiness. I end the interview and escort the candidate out. Remember, it’s not personal—there is a lot at stake for many people when it comes to getting jobs, and pressure makes people do strange and unethical things. You probably don’t want people who crack like this on your team, but neither are they necessarily inherently evil or trying to harm you.
Candidate Completely Off Track or Answers the Wrong Question
Sometimes a candidate will give you an answer to a question that is only similar to what you asked, not the question you asked. This can be unconscious; they may hope or expect you to ask a different question, or they are unknowingly using the availability heuristic.5 They essentially hear you ask an easier or different question. In my experience, all of these are more likely to happen when asking subtle variants on common questions, so patiently explaining that the question is different and repeating or rephrasing it usually brings the candidate back on track.
Still, sometimes multiple repeats and rephrasings are not enough. If the question is not critical, and you can replace it with another that informs you just as much, it may be worthwhile to move on rather than stalling on one question.
Candidates show up to an interview with their whole world going on around them, which could be in any shape and state. Your own life is no less complicated.
I’ve interviewed candidates distracted by family problems, ongoing emergencies, illness, and more. If a candidate’s life is saturated with distraction and disaster, it will certainly affect their performance, but how can you know that from one incident around the time of interviews?
When candidates indicate that they are acting under outside stress, I suggest offering to reschedule the interview to when they can give it full attention. Be sincere and understanding and the candidate will appreciate it, whether or not they accept the offer to reschedule. It lifts the customer experience and gives you a chance to fairly evaluate the candidate later.
Candidate Is Inarticulate
Brilliant engineers can consistently discuss complex topics. Candidates who cannot do so indicate a disorganized mind or dramatically underdeveloped communication skills. I do not know of any way to work effectively with a cognitively disorganized engineer.
Verbal disorganization might be caused by interview jitters or hasty, excited speech; so take some to calm the candidate and guide him or her to speak slowly and precisely.
Very rarely, you may come across a candidate you cannot understand or who does not seem to understand apparently straightforward questions and discussion. In my experience this has been caused by low capability in the interview language. Consider treating the problem in the manner described in the “Thick Accent” section. If it does not improve, consider halting the interview. You’re not learning anything.
Always interview candidates in the language you expect them to use in the ordinary course of the job.
Even an exceptionally qualified candidate may make an obvious or ridiculous mistake during an interview. Be prepared to forgive one of these; the interview situation is artificial and stressful. You make errors from time to time, too, and not always under so much pressure. Besides, errors are opportunities to learn and grow (unless they are catastrophic).
Consider the interview from the candidate’s point of view. It’s stressful and they may have a lot on the line—mortgage and tuition payments overdue, or an intense and long-standing frustration they’re trying to leave behind. Their incentive for landing a job may extend far beyond looking for more interesting or comfortable work. Nerves are quite understandable.
In addition, an interview is highly artificial situation, far removed from what you expect from developers on a daily basis. They do need to complete the interviews, of course, and you can make it easier for them. Through basic empathy, people take anxiety signals from those around them, so stay relaxed as much as you can. Be friendly and understanding, but don’t draw too much attention to the candidate’s nervousness—that can amplify the problem. Mention or address the nervous behavior and then don’t mention it again, so the candidate has a chance to calm down.
In this kind of situation, I give the candidate a win by pointing out that they are being successful, for example, in providing a key insight or solving a problem. Success seems to foster relaxation. There is an art to putting people at ease, so if this is a recurring problem in your interviews, do some research and practice.
Finally, not everyone is as nervous as they look. I’ve interviewed brilliant engineers who couldn’t stop moving and fidgeting but solved complex, nuanced problems without difficulty.
Candidate Has Strange Mannerisms
Interviews bring out twitchiness and nervous tics in candidates. It’s best to simply ignore these little quirks; interviews can be intensely stressful to some people, eliciting rare behavior. More important, some mannerisms can be caused by medical conditions (e.g., restless legs syndrome) that should not factor into your decision-making process anyway.
Candidates typically need to speak some language or another in common with your team—maybe more than one—so there’s a baseline speaking capability required. Accents are a fact of life and everybody has one, but some are more difficult for a typical listener to puzzle out.
The key to communicating with someone with a thick accent is patience. Listen intently, repeat what you have heard, and ask the speaker to restate what they’ve said. Stow any frustration; treat the communication as if it were a puzzle and you will establish a communication path.
Even an accent quite difficult to follow when you first hear it will become easy after days and weeks of exposure. Because it’s just a temporary adjustment for team members, I have yet to disqualify a candidate for having a difficult accent.
In Western cultures, silence during conversation can cause discomfort.6 A conundrum of technical interviews is that they are both conversations and tests, and the test parts demand some concentration. Not everyone can talk and concentrate at the same time—some candidates are comfortable narrating their internal problem-solving process, and some are not. A narrative gives you access to their mental states; you can see they are making progress, and you have an opportunity to provide meaningful hints and steer candidates away from unimportant tangents. Helpfully, some people even talk aloud as part of their thinking strategy.
Candidates can’t always produce a narrative, perhaps because it interferes with their thought processes or their problem-solving process is difficult to verbalize. They will quietly look at a whiteboard or stare into the distance, and time passes in silence. Just wait. If they are able to make progress, so be it. Wait a while to let them concentrate, and then you can ask what they are thinking about, whether there are potential solutions they are exploring or have discarded, or if they want a hint.
Embracing a little silence won’t hurt anyone.
After a candidate passes your initial screen, you would normally conduct an on-site interview, but sometimes you can’t. Conducting a remote interview is an option when a candidate is particularly distant or unable to travel, or you just don’t have the budget to transport the candidate to your office.
If you get to the offer stage but are reluctant to make an offer because you didn’t have a “normal” interview, then you may have wasted considerable time and energy. Before starting down that path, you must establish a bar for the candidate to meet and commit yourself to making the hire if the candidate meets that bar. If you need buy-in from your management or HR, get it first.
With a commitment in hand, use all available technology to make the remote interview as much like your on-site interview as possible. That makes it easier to evaluate the candidate in the same context as all other candidates, or as similar as it can be. Keep the process similar, too; for instance, schedule it in a contiguous block with short breaks.
If remote interviews are the exception for your interviewing team, their unfamiliarity makes it an intrinsically higher risk to hire based on remote interviews. You can offset this effect by insisting on holding to a very high standard. That will reduce the odds of a bad hire, though it does also increase the odds of missing a good hire. Such is life.
Before considering a remote interview, I look for at least two positive phones screens measured against a high bar.
Your HR department might have specific policies regarding what you may record about interviews, what you may not, and when you must destroy the records. Even restrictive policies should allow you to keep anonymized notes, in which you have stripped out the candidate’s name and other identifying information.
Use records to reflect on and discuss the candidate’s interviews, and later as an information source you can mine. In the absence of specific policy, I recommend that you write and keep for each candidate the following key memoranda: what you asked, the essence of the response, and your interpretation.
To allow you to share the exact answers with others (rather than interpretation or vague recollection), transcribe or take pictures of source code and pertinent diagrams or charts the candidate creates as part of their answers. With an exact record, you can get fresh opinions and comments from others at any time.
If your interviews are generally getting off track—if you are rejecting many otherwise apparently qualified candidates, or accepting those who aren’t—you can use records to help you find out what’s going on. For example, reading individual feedback lets you understand the consistent patterns each interviewer has established and how recap meetings affect their decisions.
By using strong statistical techniques to protect yourself from outliers and unjustified conclusions, you might discover other interesting and useful phenomena—for instance, that early morning interviews are more likely to result in a positive hiring decision.7
1 David Dunning and Justin Kruger, “Unskilled and Unaware of It: How Difficulties in Recognizing One’s Own Incompetence Leads to Inflated Self-Assessments,” Journal of Personality and Social Psychology 77(6) (1999): 1121–34.
2 Eric Sundstrom and Mary Graehl Sundstrom, Work Places: The Psychology of the Physical Environment in Offices and Factories (New York: Cambridge University Press, 1986).
3 Again, I’m not a lawyer. Review your company’s policies and relevant laws with HR if you need clarification and guidance.
4 Julia Levashina and Michael Campion, “Measuring Faking in the Employment Interview: Development and Validation of an Interview Faking Behavior Scale,” Journal of Applied Psychology 92(6) (2007)
5 Daniel Kahneman, “Maps of Bounded Rationality: A Perspective on Intuitive Judgment and Choice,” Nobel Prize Lecture, 2002, www.nobelprize.org/nobel_prizes/economics/laureates/2002/kahnemann-lecture.pdf.
6 “Brief Silence in Conversation Evokes Negative Emotions,” University of Groningen, January 20, 2011
7 Shai Danziger, Jonathan Levav, and Liora Avnaim-Pesso, “Extraneous Factors in Judicial Decisions, Proceedings of the National Academy of Sciences USA 108(17) (2011): 6889–92, doi:10.1073/pnas.1018033108.