I saw a tweet from Angie Jones which said:
less content about how to pass tech interviews. more content about how to conduct tech interviews.
I've been conducting a lot of technical interviews recently, so inspired by Angie's tweet I wanted to write about my teams process for conducting a technical interview. But before I talk about the interview process itself I want to give some context to the role the interviewees are applying for.
My team is responsible for maintaining the accessibility testing engine axe-core, and axe-core-npm which are integrations of axe-core into various frameworks and technologies. Since our team works with so many different frameworks, we expect the candidate to be willing to learn frameworks they may not be familiar with, and sometimes even become subject matter experts on them for the team.
We're also a fully remote team which means the candidate will be working by themselves for good portions of the day. This means we expect the candidate to be able to understand how to learn on their own as well as know when to ask for help. We do a lot of pairing on the team and are always willing to jump on Zoom to discuss a problem or even be a rubber duck.
We've made sure to document these requirements as part of the job description.
The Pre-interview Assignment
Before a technical interview takes place, we give the candidate a small take-home pre-interview assignment that they'll bring to the interview. The assignment focuses on configuring one of our open-source APIs to do a small task. We've made sure the task is neither large nor time consuming, and we allow the candidate a few days to complete it before the interview.
The purpose of the assignment is to ensure that the candidate is willing to learn something new (most users have not configured our API in this manner) and understands how to read at our public documentation to find the answer. It also serves as an introduction to our open-source code as the technical interview will leverage that knowledge.
The Technical Interview
My team fully understands that traditional technical interviews do not work well. We're also a remote team so we can't do traditional in-person interviews anyway. We've spent a lot of time trying to come up with an alternative method that can help us unbiasedly assess the technical ability of a candidate.
To start, we use a similar process to Lara Hogans Onsite Interview Template. We've created a list of signals that we are looking for in a candidate based on the job requirements listed in the job posting. Using these signals, we created an evaluation form that we can fill out during, or immediately after, the interview. The evaluation form is essentially a list of questions about the candidate that relate to the signals. This form helps us see at a glance how well the candidate did and helps us evaluate one candidate over another.
We've even made sure that what we evaluate for is adjusted based on the position. We expect more from senior developer candidates than mid level candidates and have created separate evaluation forms that take into account those differences. We've been really pleased with how it's turned out.
The technical interview itself is quite unique. We don't ask any technical or algorithmic questions. Instead we essentially do a pairing session (cameras off) with the candidate and ask them to work with us to solve a problem.
We provide a small, fully working boilerplate project (with tests) that has the main part of the program stubbed out. We also provide a README with the problem defined and a list of requirements for the project. The project is similar to something we would do on the job, which gives the candidate a good feel for the types of things they'll be working on if they were to join the team.
We let the candidate drive the coding session while sharing their screen. We are there to support them, answer their questions, and help them work out the solution to the problem. We very much allow and encourage looking up answers, using Stack Overflow, reading the documentation, etc. After all, we don't care so much what you know off the top of your head but more about how you go about finding answers.
The requirements are set up in such a way that there are more things to do in the project than the candidate would have time to complete in the interview. This way we can tell them that we're only interested in how they go about solving the problem, not how much they can solve.
Because this is a non-standard approach to a technical interview, we try to be very clear upfront about it. The day before the interview we send out an email to the candidate telling them what they can expect from the interview. We try to be very clear about what we'll be doing in the interview, what they can do and lookup, and the goals of the interview.
We hope that by doing this (allowing looking up answers, not worrying about how much they do, telling them ahead of time our expectations), the candidate feels less stressed over the interview and can be more comfortable during it.
Rational of the Technical Interview
There are many reasons why we do our technical interviews this way instead of say, a full take-home assignment.
First, we've discovered that doing a pairing session with a candidate can help us see their potential. Potential is something that you can't test for nor can see very easily in the final product. We once had a mid-level candidate read the requirements document and then tell us how they thought the program should be structured to make it work. Without writing a single line of code, we could tell that the candidate had a good grasp of technical design. That's not something we could have known otherwise.
Second, we want to see how the user goes about learning a new codebase. The ability to read an unfamiliar code base and understand how it works is extremely important for our team. By being able to watch them learn the code, we can see into their learning process. A side benefit is that by seeing how they learn, we can adjust the onboarding process to better match their learning style.
Lastly, we want to get a feeling for how the candidate would be to work with. Do they respond well to suggestions? How is their personality while working on a problem? How much do their try to solve the problem on their own vs having us drive the solution?
Of course, this process may not work for every team nor every candidate. Overall we feel we have succeeded in our goal to create a better technical interview process that works well for our teams needs.