Note: This blog post is based on a talk given at RailsConf 2018. Video here. Slides here. I’m available to give an updated version of this talk at your conference, company, or child’s birthday party. Contact me for details.

Let me get straight to the point: you don’t have to see a person’s code to figure out whether they’re a good developer.

Over the past ten or so years, I’ve interviewed a lot of engineers. In that time, I’ve developed a set of techniques that allow me to quickly and accurately evaluate a developer without seeing their code.

I’m now convinced that it’s not only possible, but objectively better to do it that way.

What’s a “Coding Interview”

A “coding interview” is any evaluative method that requires direct testing of a candidate’s technical skills or looking at code or pseudocode they’ve written. That includes things like:

• Reviewing Code Samples or GitHub Profiles
• Code Tests (Codility, etc.)
• Code Challeneges or Take-Home Assignments
• Live Coding or Pair Programming
• Implementing algorithms on a white board

Why Do We Do This?

These techniques didn’t come about by random chance. They all replaced far worse ways of picking co-workers. Such as:

• “clever” riddles: how many tennis balls would fit on a 747?
• language trivia: In Java, what’s the difference between an interface and a subclass?
• White board coding: Not even the worst idea on this list, somehow.
• Discovery checks: “How do you identify a loop in a linked list?”

We realized that those are all silly ways to interview. They don’t work because they don’t show any skill that’s useful in a software developer’s day-to-day job. So, we then decided to try to replicate the day-to-day job in an interview.

There’s a big problem, though: it doesn’t work. At all.

The dirty little secret of coding interviews is that they don’t actually replicate real work. 

It is true that coding interviews force candidates to show some of the skills they’ll need. But that happens in an environment nothing like the one they’ll be working in. And that’s not all.

The Problems With Coding Interviews

Coding interviews reduce the number of candidates available to us. They also lead us to disqualify good candidates and give us false confidence about bad candidates.

Coding interviews are time-consuming for both the interviewer and the candidate. The former would be okay if it produced good results. The latter makes it difficult to hire good engineers in a candidates’ market.

It can be difficult for you to keep the coding assignment up to date, leading to bad interview experiences. Many passive candidates who are happy at their jobs aren’t going to do your code challenge at all. They also increase the length of your interview and the length of your hiring process. None of those things improve the volume of quality candidates you’ll see.

What’s more, coding interviews aren’t actually good analogs for the real work.

Google research found that their most successful teams displayed a trait called “psychological safety.” The teams performed best when they felt comfortable taking risks and being vulnerable.

This isn’t at all compatible with a coding interview. Candidates are likely to feel anxious or stressed, worried that one wrong move might cost them the job. You’re not getting many people’s best work.

As if that weren’t enough, these sorts of practices disadvantage people who don’t have a lot of free time. Using them will cause your hiring process to surface the people who do. Your candidate pool will tend to be younger, whiter, better off, and more male than it otherwise would be. You’ll also miss out on other groups, like single parents or people taking care of sick family members.

What’s a Better Alternative?

I hate to be the bearer of bad news here, but you’re going to have to talk to your candidates. You have to learn about them. You have to carry on a conversation. It may not be easy, but it is simple.

You have to develop your skills as an interviewer.

Interviewing Skills: A Prerequisite

Coding interviews have an air objectivity, while being anything but. They’re a poor replacement for thinking hard about what skills and traits are valuable to your team.

Hiring is hard, and doing it well could be an entire series of blog posts unto itself. But, there are some things you can do to instantly level-up your interviews:

• Define what you’re looking for in a candidate ahead of time. Be specific and detailed.
• Ask your candidates a consistent set of questions.
• Write down all interview feedback. Do it right after the interview and before discussing with anyone else.
• Practice, practice, practice.

Code-Free Techniques

There are probably a lot of ways to evaluate a candidate without making them code. The three techniques that I use cover an overlapping (but mostly different) set of skills. During an interview, we talk about their experience, ask them to review some code, and design a system collaboratively. Each is detailed below.

With these techniques, I’m trying to find candidates who can do the technical work while being a valuable addition to the team outside of their raw programming skill. In a typical interview, I will cover all three of these sections in around an hour. That’s almost always enough to feel confident that I got a good read on the candidate.

Dig Into Their Experience

Most teams are already doing this. They spend a few minutes on it at the beginning of the interview. They ask what the candidate worked on, how they liked it, and so on. Most of the time, it feels like small talk.

That’s the wrong approach.

Remember that this is an interview. You want to understand the tech they’ve built to the greatest degree possible.

To do this well, you need to have read their resume before you get into the room. No kidding. Block off at least 10 minutes before you need to walk to the room to read (not skim) their resume. If you can block off 30 that’s even better. Learn as much as you can about their projects from their resume. Google to see if you can learn about their projects from public sources. The less background they have to spoon-feed you, the faster you can get to the good stuff.

In the interview, ask them to tell you about the most interesting thing they’ve worked on recently. Practice active listening. Be engaged. Pretend you’re joining their team or that you’re doing an architecture review. You’re trying to learn what they built and how. What are its strengths and weaknesses? Make sure the candidate knows that not knowing an answer is not a big deal, but bring your curiosity.

Here are some questions I’ve found to yield good answers:

• What was your role on this project? — This, alone, is not determinative. Even someone with a minor role can be a good fit. Your candidate may be looking because they didn’t get a large enough role. It still helps to know what their role was.
• What support did you get from other people?  — Being unable to identify support is a huge red flag. Even in solo projects, there are almost always folks who helped. You don’t want a self-centered co-worker.
• Walk me through how that feature works. — Explore where the data comes from, where it goes, how it’s stored, and what benefit that all provides to the end-user. This question is a jumping-off point for your natural curiosity.
• What’s the worst technical debt in this project? — An engineer that understands the tradeoffs they’re making is better than one who doesn’t. Follow up by asking how they’d fix it if they could and why the team hasn’t decided to fix it yet.
• Has it had any production bugs or outages? — See if they understand what caused the bug and how the team resolved it. Should they have anticipated it? What could they do next time to avoid this problem?

This section of the interview helps you learn about the candidate’s direct experience. Done well, it can also help you learn how they share credit or place blame. You will learn how they think about tough engineering choices. They’ll share some lessons they’ve learned recently. Their ability to communicate about technical topics to relative laypeople should be obvious.

Be prepared to hop to a different project if they picked an unsuitable one. This could mean it’s not complex enough or they don’t remember it well enough.

Caveat: In this section, be careful with questions like “Tell me about a tough bug you solved.” Asking someone to recall the specific mechanics of an ephemeral part of a system can produce a lot of false-negatives. Not everyone carries around knowledge of all the bugs they’ve fixed. Putting someone on the spot for this kind of info is a good way to inject a lot of stress into the interview.

Have Them Review Your Code

This exercise is 50% code review and 50% role-play. It’s a chance to identify candidates who can level-up your whole team’s code and make your office a better place to work in the process.

Here are some areas of focus during the code review:

• How do they interact with the code’s “author”? Is it helpful? Productive? Kind?
• What sorts of problems do they emphasize? Do articulate their concerns clearly? Do they jump right to bike-shedding on style issues?
• Are they comfortable reading unfamiliar code in the language you’re using?

This method takes a bit of setup. You’ll have to find or write the code that your candidates will review. You’ll also want to create a priority-ordered list of the issues with the code that you want them to find. Do not leave this to the interviewer’s discretion; decide ahead of time.

When selecting code to review, avoid production code. Your candidates won’t have the context that you will. You’ll be tempted to compare them to your co-workers rather than to other candidates.

Work to reduce complexity in the code sample. They won’t have a lot of time to read it and they likely won’t be in a code review mindset at the beginning. It may take a bit of warming up.

Include a realistic bug or two, but don’t emphasize bug hunts. Code reviews, in general, are a terrible way to find bugs in code. This is doubly true if the reviewer has never seen the code base before. Self-evident bugs¹ are best. Bugs should almost always be at the bottom of your priority list. They should function more as a bonus for extra thorough people.

Finally, the code should be doing something accessible. If your company is well-known, it could be a stripped-down version of your product. But, don’t do that if you’ll have to spend time explaining providing a lot of context.

Your best options are either made-up code²  or an open source repository with a straight-forward pull request.

Once you’ve decided what code to review, here are some things you might expect a candidate to find:

• Bad pull request descriptions or commit messages
• Working but non-idiomatic code
• Overly complex code (things in need of refactoring)
• Confusing variable or method names
• Overly architected classes (YAGNI)

If the code, as written, doesn’t contain enough problems, add some more.

There’s a hidden question here that I haven’t quite landed on the answer to, yet. Should you send the code to review ahead of time?

If you do, you’re back to advantaging groups that already have a huge advantage: those with a lot of free time. If you don’t, you run the risk of injecting a lot of stress into the situation.

I lean toward the latter option. A good interviewer can mitigate the stress. One way to do that is to let candidates know ahead of time that they’ll be reviewing code. You’ll also want to set clear expectations at the beginning of the session.

Design a System Together

One of the most important jobs you’ll have as an interviewer is determining whether the candidate can do the technical work you’re hiring them to do.³

A good evaluative technique is one that minimizes false positives and negatives while also allowing you to differentiate between a wide range of skill levels.

The most junior candidate should be able to make some progress on the problem.  The most senior candidates should run into problems they can’t solve instantly. It should get to that point quickly.

I’ve used this technique on college and code school students as well as 20+ year industry veterans. It works well across the board.

What is it?

Collaborative System Design is working with your candidate to architect some tool, platform, or project. There’s no code or pseudocode. You talk about how you’d design the system and what trade-offs you’d make. You also propose potential issues those trade-offs cause and how to adjust to them.

This is effective because the difference between a Junior Engineer and a Senior Engineer is, primarily, the number of mistakes they’ve seen. That experience translates into more robust system designs and an ability to solve new problems on the fly.

How Does it Work?

Your team picks a project they want the candidate to “build”.⁴ Time-box the exercise. Let the candidate know how long you’ll be spending and that you won’t finish the whole thing. They should know that’s the expected outcome.

Spend a minute or two setting up the exercise and explaining the basic requirements. If you can’t explain the system in a couple of sentences, it’s probably not a good subject. For example, your project might be building a clone of a well-known social media platform. You shouldn’t have to explain everything about how Instagram works, for example.

Supply any parts that you feel might be a red herring. Then, ask the candidate what they’d build first and how. You continue discussing new features until you run out of time.

If the candidate seems like they’re sailing through it, increase the complexity. If they are struggling, reduce it.

The variable difficulty allows you to hone in on the candidate’s skill level. It’s like binary search for the candidate’s level!

Picking a Project

The key point is that the requirements should be easy to understand. If your company’s product is well-known, it can be nice to use that. If not, use a commonly known site (like Facebook). The project should be related to the work the person will do. We have one project for our back-end hires and one for our front-end hires.

Our back-end project focuses on the underlying systems of a popular social network. Our front-end project takes a stripped-down screenshot of one of our product’s features and asks them how they’d build that (what are the components, how is interaction modeled, etc.).

Sample: Building Facebook

First, we lay out the task. Tell the candidate we’re going to be building a Facebook clone. We pretend that we have a running app that contains users, authentication, and little else.

At this point, we ask our candidates what they think would go in a Facebook MVP. We like this because it lets us see how our candidates think about product needs. It also gives context for decisions and trade-offs they’ll make later on in the exercise.

At this point, I like to get up to a whiteboard and write down their thoughts. This keeps me engaged and it ensures I’m understanding their design. If I get something wrong, they’ll know immediately and be able to clarify.

Our candidate might tell us that our MVP includes friending, status updates, a news feed, and likes. Ask a couple of questions about how they came to that feature list. Is there an obvious feature left out? A superfluous one kept in? The feature list isn’t as important as their thought process. Then, ask what they’d build first and how. Let’s say the candidate decides to build out the “friends” functionality first.

Right off the bat, they have options. Some candidates are going to come up with solutions that might not fully work, but they shouldn’t be lost. Your most senior candidates at this stage are going to dispense with this pretty quickly.

Pretend our candidate tells us they’d create a table to hold friendships. That table would contain columns for friender and frinedee.

Start digging. In Facebook, friendships are bi-directional. How do we represent that in our database? How does that impact our design?

One option is that we don’t need to do anything special. To find a user’s friends, we check both columns. Another option is to create two entries in the friends database — one to represent each direction. If we do that, we’ll have to delete both if someone gets un-friended. Discuss the tradeoffs.

The next topic of discussion could be how to handle friend requests. They might choose an “accepted” flag on the friend row. They might create a new table for friend requests. Next you might talk about blocking or spam friend requests.

At some point you’ll run out of depth on friending and you can move to the next feature. As new features get added to the system, you can have a lot of fun talking about how they interact.

If your candidate gets stuck, that’s fine! You can propose a solution and ask them what they think. What might be good or bad about it? If they seem like they’re way over their head, it might be time to discuss a different feature.

One of my favorite things about this exercise is that it can be fun. If the interviewers do a good job and the candidate is engaged, you can end up with something that doesn’t feel much like an interview. It will feel like a conversation you’d have with a co-worker about a new project you’re getting started on.

Which is exactly the role the person is auditioning for. Perfect.

When Should You Use a Coding Interview?

As much as I hate them, they still have their place in a few limited scenarios:

• Determining if entry-level candidates can program at all.
• If your organization mandates lots of pair programming, pairing in the interview is a good preview of their future work.
• If a role might require a very specific programming skill and you won’t have time to teach it. This is especially true if you’re on the fence between two seniority levels or titles.
• When you’re transitioning from a coding interview to a code-free interview. Don’t quit cold turkey. A transition period can help you build trust in the code-free techniques.

Finally, you might want to use a coding interview if your team’s interviewing skills are weak. Coding interviews are less effective and have a lot of downsides. But, to their credit, they are considerably easier to run. Inexperienced or unsure interviewers may feel more confident with them.

If this is you, developing interviewing skills should be a top priority. Hiring the wrong person is an expensive mistake.

Conclusion

With a little practice, your team can get really good at this style of interview. They’ll be able to make better hiring decisions in a shorter amount of time, and your interviews will be accessible to a much wider range of candidates.

I hope this has been helpful. I’m always happy to take questions or feedback over Twitter or via email.

Good luck eradicating coding from your interviews!

Header image by Rawpixel on Unsplash.

---

1. E.g. passing a string into a method that requires an array. ↩

2. Maybe from the code challenge you’ll be getting rid of after reading this blog post? ↩

3. It’s not usually the most important. Their communication and teamwork will usually have greater impact. ↩

4. Every candidate should have the same problem to work on. Over time, your interviewers will get better at evaluating responses. ↩