Hiring in software development is horribly, perhaps irretrievably, broken. As both a software development manager and a software developer, I’ve seen the process stop good developers from getting jobs while incompetent numbskulls who shouldn’t be anywhere near production get hired over and over again. The latter do so only to get fired a few months later when they can’t deliver results. Usually the rest of us get to clean up their results for months or years after they are gone. It’s really dumb that most places use a broken process, but that’s the reality we have to deal with.
In this chapter, we’re going to discuss the reasons why it is difficult to evaluate software developers, how companies have traditionally tried to handle it, and how various approaches have failed. Understanding why things are messed up is very helpful in dealing with the problem in a constructive way, even if you can’t fix the problem itself. Software development hiring managers have been trying to find a good way to evaluate developers for decades, and have yet to come up with anything that both works well and is trivial to implement. We’ll discuss how the industry first used quizzes, live coding challenges , and various other approaches to evaluate developers. Then we will discuss how the industry settled on using whiteboard interviews as a common way to evaluate developers.
Difficult to Evaluate
The biggest problem of the software development process is that it is difficult to evaluate candidates for a variety of reasons, especially when the market is tight. If you hire someone unqualified, they can (and will) easily cost you thousands or tens of thousands of dollars before you get to go through the process again. All that is assuming that they don’t take the company down with them, putting you in a position where you are interviewing from the other side of the table. On the other hand, if you do manage to get the resume of a good developer across your desk and the market is full of jobs, you may not get a lot of time to decide whether you want to hire this person or not, before they get snapped up by another company in town. Hiring managers are in a very difficult position and are thus forced to make a lot of decisions quickly, even if they come to regret those decisions later.
Worse still, hiring managers don’t have a lot of time. If a healthy company is hiring, it’s because they either have more work than they can do or they are anticipating having more work than they can do. Neither situation gives the hiring manager a lot of time to carefully consider candidates. This tends to mean that they will develop shortcuts to cut down on how much time they spend on the hiring process. This is especially true of software development managers who are still writing code themselves. Many software development managers don’t enjoy managing people and will go to a lot of effort to limit their time doing so. As a result, you are very likely to be interviewed by someone who is in a hurry, doesn’t particularly want to be there, and just wants to be done with the whole thing so they can get back to the rest of the work on their plate. This state of affairs can be a huge disadvantage to the unprepared, but can be wonderfully helpful to those who have done the work up front to make it an advantage.
Software development is unusual, especially for such a high-paying profession, in that there are a ton of ways that someone can get into the industry, with varying degrees of success. Not all software developers go to school for it; in fact, on the majority of teams I’ve worked with, most developers haven’t had a computer science degree. And it’s not like such developers lack skill—my friend Jeremiah doesn’t have such a degree, and I know he can code circles around me. The difference is made with diligent practice, work ethic, and experience. The degree can help, but is not a sufficient measurement on its own.
And there’s another problem too. Not everyone that goes to school is good. In fact, I’ve worked with a number of people who have gone as far as getting a PhD in computer science who, being kind, can’t code their way out of a wet paper bag. You can’t use a degree as proof of skill in software development, because it isn’t one. Many companies try to use a computer science degree to evaluate people and often end up passing up on candidates that would have been an excellent fit for them. I’ve seen good candidates ignored and bad candidates hired, solely on the basis of a degree.
Yet another issue occurs due to this lack of focus on credentials and high pay. These conditions often attract people who wouldn’t be pursuing this field if it weren’t for the pay. As both a development manager and a developer, I’ve interacted with dozens of such people, including a very scruffy 20-year-old who was loudly certain he could get a job paying $70,000 a year because he “knew” how to build web applications. As it turned out, what he had done wasn’t any more complicated than what you’d expect out of someone who had just had an introduction to HTML and had been halfway asleep while getting it. I guarantee you that any experienced hiring manager can tell dozens of similar stories. Not only do they have to evaluate the top people, but they have to filter this sort out before they waste too much time.
Busted Problems, Busted Solutions
Since the 1990s, when learning how to code became far easier and accessible, the industry has tried a variety of hacks in a vain attempt to quickly evaluate developers and get around these problems. All of these attempts had issues that meant that they didn’t live up to their promise. However, by examining the shattered wreckage of these bad ideas, we can learn a lot about what kinds of things hiring managers are trying to determine in the space of a short interview.
The Development Quiz
When I started trying to get paid for writing code, most development shops were trying to solve this problem with quizzes. Because it was the 1990s and early 2000s, the Internet played far less of a role in these processes initially, since it was slow and hard to search, and web applications were very limited. Essentially, the idea was that you would be presented with a printout with a series of questions about your chosen platform, and your answers to those questions would essentially be used to evaluate your suitability for a job. Many places developed their own questions in house, based upon the needs of the business. There was a short period of time where this worked really well, because the hiring manager’s questions for the developer came right out of their own experience and reflected the kind of problems that one might encounter on the job.
However, it wasn’t going to last. As the dotcom bubble approached and passed by, two things spelled the end of the in-house development quiz. The first was the wide availability of decent Internet search engines (stuff like Dogpile and Lycos) which allowed hiring managers to simply search for quizzes to give, rather than carefully formulating their own. However, the same technological change also allowed savvy developers to look up the same quizzes and prepare ahead of time. Between about 2002 and 2008, I bet I saw the same quiz in at least a dozen interviews on C#. As a result of these changes, the value of the in-house quiz became lower by the day, becoming roughly useless in a matter of months. This resulted in companies abandoning the practice within a few years.
Such a scenario is quite bad for developers, because an environment where everyone has the same answers to the same questions is an environment where you can’t differentiate yourself as easily. You may see a few companies still using this, but it isn’t as common since people figured out how to hack it. If you do see this in use, you can learn a lot by comparing the questions they ask you to those that are commonly available on the Internet. If they’ve carefully formulated questions that are specific to their needs, that’s a good sign. If the list of questions is one that you’ve seen before, then they aren’t being as careful as they probably should be. However, if you can do well on this part of the interview, it will still help you.
Development quizzes offer a few advantages to the interviewer. First, it is an approach that scales well, so they can give the same quiz to a large number of people without greatly increasing the amount of effort they are expending. Secondly, this approach allows nondevelopment managers to use the quiz as an initial screening mechanism . You’ll often see this when you have an initial “phone screen” with Human Resources personnel—the quiz is intended to filter out a large percentage of candidates so that the hiring manager can choose among the best. Finally, such a quiz is low maintenance. Whether a company develops their own or gets one from the Internet, they don’t really have to do much to maintain the quiz. It will generally work as well in a year as it does today.
Because of these advantages, development quizzes are still something that many hiring managers have in their arsenal. However, it was soon to be superseded by other approaches that were a little harder to game.
Coding Challenges
Another approach that became common later on was the coding challenge. While a few places let you take the challenge home and bring it back in, most actually required you to sit at a computer on-site to complete the challenge. The take-home challenges tended to be more difficult, while the in-house ones were simpler. There were a lot of upsides and downsides to both approaches.
Take-home challenges offered a great way for companies to evaluate a lot of developers concurrently. Instead of tying up the team while someone coded, this approach allowed companies to screen developers in a way that scaled pretty well. If the developer didn’t complete the challenge, they were no longer in consideration. However, this approach had a lot of problems. First, and most obvious, it made cheating easy, whether such cheating was done through one’s own circle of friends or via the Internet. Second, many developers balked at these challenges, feeling that they were essentially being asked to do work for free. Finally, while the practice was very scalable for hiring managers who wanted to test out a lot of developers, it didn’t work very well for job seekers who wanted to apply at many positions. Once a developer has completed a take-home challenge or two without being hired, they are more likely to avoid them in the future.
A few companies tried to do coding challenges in house, but there were problems with this approach as well. First, the challenges couldn’t be very large, because it actually wasted the company’s time (instead of “just” the developer’s time). Second, and perhaps most important, it also meant that a random stranger off the street was going to be using a computer inside the corporate network. This either required a lot of preparation up front or constituted a risk that most companies weren’t willing to take once they thought about it a little.
Live coding with people watching you is almost as bad as a whiteboard interview for many of the same reasons, but it probably would be an improvement. Nevertheless, this approach is not something you are going to see much anymore. That said, if you do see a company using this approach, it’s potentially a very good sign, as it means that their pool of applicants is smaller. Competing against fewer people is often easier, so long as you can compete effectively.
Whiteboard Interviews
As a result of all this experimentation, the industry ended up with the much-feared method of programmer evaluation that is a major subject of this book. This process came about for a variety of reasons that seemed like a good idea at the time.
First, interviewers are pressed for time. It’s uncommon for companies to hire when they don’t already have a need. As a result, the people involved in the interview process probably already have more work to do than they can expect to complete. The whiteboard interview theoretically offers a pretty reasonable way for them to evaluate developers without spending a ton of time, giving them the opportunity to cheat, or giving them access to computers inside the company network.
It’s also very common for interviews to include an interview with the team to “assess cultural fit,” which is an HR euphemism for the process of figuring out whether you and the team can stand each other. These usually happen in conference rooms because of seating constraints. This process will be discussed shortly, but the basic idea is to seat everyone together and discuss why you might be a good candidate for the job. Afterward, the team will be consulted to see what kind of impression you made.
As a result of these things, along with the near certain presence of whiteboards in conference rooms, the whiteboarding process was a natural fit for the way many organizations were already doing things. Add to that the ability to quickly find sample whiteboard problems on the Internet, and it totally makes sense why this happened. Whiteboard interviews have thus become very common, and are often the first choice, especially for stressed-out hiring managers.
All this history is important, not because you are going to be given a test on how interviewing practices have evolved but because the history itself points to the kinds of problems that interviewers are trying to solve using the whiteboard. If you know those things, it’s easier to make appropriate decisions about what to do based on what the interviewer is actually looking for.
A whiteboard interview also tells you a few things about the company. First, they have a pretty standard approach to software development hiring, and they probably (mostly) interview average developers. Depending on your career situation and goals, this can either be great news or can convince you to look elsewhere (we’ll discuss this in a later chapter). While it may not sound so great for a company to just be interviewing “average” developers, there are a lot of reasons why you might want to work at such a company.
Evaluating Soft Skills
A few things also make coding skills even more difficult to evaluate than normal. While you as an interviewee may not be concerned about these things, you might want to start thinking about how to use them to your advantage. In this section, we’ll discuss things that make interviews even harder for the interviewer. Later, we’ll discuss how to turn them to your advantage as an interviewee.
First, interviews and live coding exercises done remotely are often more difficult to evaluate. You lose a lot of useful information when the other party in an interview is remote. Besides all the technological problems that will invariably occur during such an interview, you also tend to miss most of the body language that the other side is using. Body language is a significant portion of our communication, and there is a greater chance of miscommunication when it is absent. At best, miscommunication makes interviews more frustrating and longer. At worst, it can completely derail the entire interview. These things are true for both parties in the interview.
Second, interviews involving a language or major cultural barrier are more difficult. This doesn’t mean that you should avoid interviewing somewhere where the interviewers speak a different language or come from a different culture. Rather, it means that communications will be more difficult. Besides miscommunication from language barriers, cultural barriers can also make things more challenging. Even cultural differences within the same country can represent large differences in manners, ways of communication, and even ways of expressing yourself. These problems easily result in miscommunication, as well as being possible sources of bias.
Third, interviews conducted by nontechnical personnel create a lot of challenges. Besides simply not knowing enough about the technology being used, nontechnical interviewers will often fixate on the things that they do know, even if those things have nothing to do with the job in question. While this doesn’t necessarily work to your advantage, it does mean that the interviewer may not be able to answer many of your questions. This can make it more difficult to decide if the job is worthwhile.
Finally, if the interviewers don’t agree on what constitutes a good candidate, it gets much more challenging to be perceived as one. This can happen because of a lack of clear organizational structure, a “hybrid” job, or because of internal politics, but it will make the interview more difficult no matter the reason. You may find that some of your answers please one party and irritate the other. It’s a difficult situation, made more difficult because you don’t understand the situation that caused it.
Characteristics of Good Evaluations
Many companies are also terrible at evaluating soft skills. While your ability to work well with people is often given less emphasis for technical jobs, it is still critical to the success of any project in which you are involved. There are a lot of reasons that companies have difficulty with evaluating soft skills.
The first (and simplest) reason that companies aren’t good judges of soft skills is that at least some percentage of job candidates are dishonest. Developers command high salaries, and this money provides incentives for less-than-scrupulous people to act deceptively in interviews. Whether they are lying about previous situations, providing an inaccurate view of how they would really handle a job situation, or simply relying on flattery to get through the interview, the results are often the same. Six months after hiring such an individual, the company is often left trying to figure out how to fire them without getting sued. Even worse, such an individual can cause so many interpersonal problems in an office that the best staff members leave.
Secondly, interviews are usually short by design. Neither the interviewer nor the interviewee can afford to have their time wasted. It takes a while to get a good view of someone’s personality, and such extended conversations are difficult in an interview. Many people with truly bad soft skills and interpersonal habits can come across as charming, fun people when you first meet them, and only become an obvious problem later when things start falling apart.
Third, interviewers often can’t get a complete picture of a person’s personality even by contacting references. It’s especially difficult to find out about character flaws from references, either because they don’t know or refuse to tell you. There is a legal risk to disclosing negative information that keeps someone from getting a job. While a lawsuit is not particularly likely, many companies are justifiably paranoid about giving any negative information about anyone when a stranger calls for a reference. It’s fairly common practice, especially with larger companies, to only confirm that you worked for them and when—they won’t tell anything else at all.
Finally, work habits are often very tricky to evaluate. Take, for instance, an employee that is frequently late to work. Are they late because they are inconsiderate and hate their job? Do they have a short-term family situation that is making it impossible to arrive on time? Are they just disorganized? The first case might get better if they were in a more satisfying job, while the second might get better on its own. The third situation might get better with time and maturity, or it might not… An employer might be more or less willing to work with someone in each of these situations, but they may not even know that punctuality is a problem for weeks or months after hiring someone.
While soft skills are critical for being a successful software developer, they are one of the most difficult and risky things for an employer to evaluate. The damage caused by employees with poor (or nonexistent) soft skills can last for years after those people are gone. In a later section, we will discuss some things you can do to make sure that you make a good impression.
Other Methods of Evaluation
While whiteboards are almost certainly the most common way to evaluate developer skills, there are a few other approaches that are better. Typically, these approaches attempt to combine the best of earlier approaches with good techniques for determining culture fit. You won’t see these often, but they frequently are a sign of a forward-looking company.
First, a good evaluation method will show your problem-solving skills on a real problem. This means that it will show your design skills, your debugging skills, and your ability to deal with ambiguous (or even changing) requirements. Such an approach should show off your ability to ask relevant questions and show how quickly you can understand the underlying business requirements. This shows that you can be a useful team member, who doesn’t simply lock themselves in an office and write code.
Second, a good evaluation method will show how you interact with a team. It should incorporate giving and receiving feedback. It should show your reaction to corrections and suggestions. It should also show how you interact with team members and whether you can do so in a healthy manner. This shows that you won’t upset the existing team dynamics (if healthy) or, at the least, won’t make them worse (if unhealthy). It also allows you to evaluate whether you can stand to work with these people.
Third, it should show that you can actually do the work. While many people think that the purpose of an interview is determining whether you can do the work, it is not the only consideration. However, you do need to show that you can actually get work done. A good evaluation should definitively prove that, without it being a matter of guesswork.
Fourth, a good evaluation process should give you an idea of what it is really like to work at a company. While you can get hints of that in an interview, the day-to-day grind of many jobs is something you have to experience firsthand to understand. Many companies have processes or just general ways of working that make them unpleasant. These seldom come up in interviews, but most are pretty obvious within a month or so of starting.
Given the previous points, one excellent way to evaluate developers is to actually hire them on a contract basis for a small chunk of work. Not only does the developer get exposure to the team, internal processes, and the problem space, but they do so for a longer period of time than a standard interview. If done correctly, both parties will know whether they can work well together (or not) by the end of the engagement. While this process doesn’t scale, if prior screening mechanisms work well, it can often produce better results than an interview. I’ve worked at several companies that have done this and they have been some of the best ones. Not only do you get a real feel for how the company works, but you also get enough exposure to potential coworkers that you can truly tell whether you’ll get along with them or not.
Summary
It is difficult for many companies to hire well. Whether it is because of the general difficulty of evaluating a candidate’s suitability, various situational problems that complicate the process, or simply because evaluating a candidate’s personality is simply a hard problem, most companies find the process difficult. While these all sound like problems for the employer, they also represent an opportunity for the interviewee. If you are conscious of the things that make interviewing difficult for the interviewer, you will have many opportunities to stand out from the rest of the candidates. Most people go into interviews thinking only about themselves, their skills, and how they are going to get the job. If you go in instead thinking about the problems the interviewer (and the company) is facing, you will make better decisions during the interview. When you are deeply aware of the problems faced by others, you can often find ways to work together to create a better situation for everyone involved, and this extends to the process of interviewing.