Colin Howe

30 May 2018

Technical interview exercises

I’ve interviewed a fair number of software engineers, hired a bunch and had to say goodbye to a few due to my own mistakes in hiring. As a result of this I have landed on an interview setup that works for me, my teams and (perhaps most importantly) the candidate.

This is a post about how I run the technical interview after a phone screen has established that the candidate is looking like a potential fit for us. I will likely follow this post with another looking at cultural fit, growth potential etc.

It’s an engineers market

One thing to bear in mind when hiring engineers is that it is generally an engineer’s market. There are more good jobs than there are good engineers. You’re in competition with a lot of other companies and so you have to try and make this process as rewarding as possible for the candidate to ensure that they stay interested in your company.

Even if it was not an engineer’s market it is always worth making the process decent for the candidate. They’re another human being and interviews can be some of the most stressful times for someone.

Why we interview

Before getting into this it’s worth taking the time to think about why we interview. There are a few things we’re doing:

In my experience a lot of energy is spent on thinking about company/candidate fit and not as much energy is spent on thinking about role/candidate fit and everyone does the same standard exercises that lead to mis-hires.

Role/candidate fit

Let’s take a look at what a typical software engineer might need to do in a role:

It’s a long list and you can’t interview for all of these skills, otherwise, your interviews will be either very shallow or very long. On top of this, if you spend a lot of time interviewing for skills that aren’t used very often in the role (e.g. performance optimisation) then you’ll be giving a false impression of the role to the candidate.

Pick a few skills that matter (3 or 4) and interview for those in as realistic a way as possible - setup 30 minute exercises that actually demonstrate the skill. This not only helps you establish skill levels, but, allows the candidate to see what kind of work they might be doing.

The most important skill that I haven’t mentioned is learning as this runs through all of the other skills. Throughout the interview I am looking for people that can learn quickly.

As well as learning, the top four skills I focus on are: architectural design, testing, coding and debugging. These are the bread and butter for what most engineers I hire will be doing.

The exercises

Introduction

All of my interviews start with an introduction. All of the interviewers introduce themselves, a bit of their background and their role in the company. We then ask the candidate to do the same.

We then explain how the process works, what the exercises are and what we’re looking for. I always explain the following:

We then reverse the standard process and ask if the candidate has any questions at all and that they should always feel free to ask questions throughout the interview. We won’t judge any questions, but, we may refuse to answer the question if it’s going to obviously give them the answer.

My architectural design exercise

Architecture is a very broad topic. As a result I have tailored this exercise to the startup environment that I operate in.

Typically, this exercise is a whiteboarding session (paper is also offered if the candidate prefers). The interviewers then say something like:

We’re a group of entrepreneurs and we’ve come to you as a technical expert to build us a system that allows users to do X.

The X is normally some kind of system that allows people to interact with each other in some form. For starters we’re looking for people that ask questions and attempt to gather requirements in some methodical manner. From here, candidates typically fall into two groups: breadth first or depth first. Both are fine.

We’re looking for candidates that build a simple system first and then we start throwing spanners in the works. What happens if we need to cope with X thousand concurrent users? What if the database blows up? What about hackers?

The best candidates will generally build a simple system and adapt it as we throw in more requirements. They will explain their decision making process (or we ask them and they can explain). The best candidates will allow for decisions to be made for human reasons - e.g. I would probably write this all in Python as that is what the team knows and it’s only a simple web app.

The testing exercise

This is another whiteboard exercise:

You work at NASA and you’ve received a program. The program runs on the terminal and does Y. As this is NASA we really need to know that Y works and won’t blow up our space ships. You don’t know what language it was written in, you just have the binary. What kind of test cases do you think you need?

Y is normally a really simple task like counting words in a string or something. We typically then write up a sample test case to get them going.

The best candidates will do a few of the ‘happy’ cases and then start thinking about edge cases. Their prior experience will often shape what kind of edge cases they think of (e.g. people with a history of C will start trying to make buffers overflow). This is normally a great chance to throw in some curve balls and get a chance to demonstrate learning. E.g. you might start talking about floating point precision and how that might break the program.

The coding exercise

The coding exercise is normally done as a test-driven development ping-pong exercise. The task is to build the program that was being tested in the previous exercise.

The best candidates will get into a good flow of writing simple code, getting tests passing and then refactoring. The best candidates will explain what they’re trying to do and write good clear code.

The debugging exercise

I was nervous when I first added this exercise to my standard set. It’s a bit naughty. I normally write a simple program that has no bugs in it and then I tinker with some underlying library, or OS configuration (for devops roles), and introduce a bug (e.g. I tweaked the MySQL driver to only ever return odd numbers when doing a count). This exercise is introduced as being a simple program that has been written and then a bug has been created somewhere in the system.

The program being debugged is intentionally very simple and looks something like the following pseudocode:

db = get_connection()
db.table.delete_all()
db.table.insert([
    'bob has %d friends' % (i * 2)
    for i in range(200)
])
print('My program inserted %d records' % db.table.count())

This is a heavily coached exercise and it generally only takes 20-30 minutes for someone to figure out what is wrong (90% of people get to the right answer). We make it very clear that we’re more interested in how they approach this particular problem than whether they actually get to the root cause or not.

The full (modified) source code for everything is available to the candidate, as are debugging tools. We also encourage modification of the code and use of other utilities to verify things if that is what they want to do.

The best candidates will do a binary search to establish where in the program the bug is happening. They will then come up with some hypotheses and attempt to prove them. They will try and verify behaviour using other tools (e.g. use the standard MySQL client). The best candidates will look up documentation, but, be open to the documentation being incorrect. They won’t be frightened to dive into the source code for lower levels or open a debugger.

Informal chat

After the main exercises are complete we get someone from outside the process to have an informal chat with the candidate for 10 minutes whilst the interviewers answer the question: do we need to know more or can we make a decision? We’re not trying to make the decision here, just establish if we know what we need to.

Wrap up

We then give the candidate another chance to ask questions before letting them know we’ll be in touch in the next day or two with feedback.

Post-interview

After the interview each interviewer then immediately scores each exercise out of 5, writes up any good/bad things they noticed and then decides their overall verdict. Once this is done the interviewers get together and do a simple thumbs exercise. After a count of 3, give either two thumbs up, 1 thumb up, 1 thumb down or 2 thumbs down. We record this result for future analysis. Then the hire/don’t hire debate begins!

It’s important that all of the scoring and decision making is done independently to avoid anyone influencing anyone else.

Once a decision is made, feedback is given as quickly as possible (typically the same day or next day). Giving high quality feedback quickly will give the candidate a really good impression of you - even if the answer is a no. In the past I have rejected people for a role, but, have received indirect feedback that they really enjoyed the interview and learnt a lot. This is the result I’m aiming for.

Conclusion

I think I’ve landed on a pretty good process. It has reduced the number of mis-hires I’ve made and has also enabled me to make a few hires I might not have made otherwise (people that I know would have done poorly in interviews that some other companies run).

I’d love to hear thoughts and feedback on this. It’s definitely something that I continue to try and improve!

Discussion