Colin Howe

My Management Style

Fri, 01 Mar 2019 13:00:00 +0000

I thought I would take a break from blogging about being a startup CTO and instead answer a question I often see asked: “what is your management style?”.

1. Trust is everything

I operate on the basis that I absolutely trust everyone on my team and I want them to trust me. If this isn’t happening then either the trust needs to be improved or the individual isn’t right for my team. Trust is an overused term and I think it’s worth defining what it means in this context:

Seeking help/guidance when appropriate and acting on it
Making decisions that are ‘good enough’
Be honest and open about reasons and motivations
No gaming the system
Having the right safety nets in place to allow for mistakes to happen safely

Importantly, this works both ways.

If trust isn’t there then everything gets slower and less joyful. I’ve not yet met anyone who relishes being untrusted. I’ve seen plenty of cases where tasks take longer because people are covering themselves because of a lack of trust.

2. Collaborative, not lazy

I don’t know everything. My team probably doesn’t know everything either. I like to work collaboratively with my teams. I believe that being collaborative makes it easier to get buy-in for decisions and helps people understand why those decisions have been made.

I feel it is important to clarify that this isn’t an excuse for laziness. If I’m wanting to change how we do something as a team then I’ll do my research and come up with a reasonable starting position that we can then work on together (or scrap if it turns out to be terrible).

3. Hands off

As I trust my team and we have worked together to make decisions I like to be hands off and let people work in whatever way suits them. I’m always available to help and talk about things. I’ll try to spot when my team is in trouble and too deep to realise they need help. Other than that, I’ll let my team get on with it with minimal reporting.

I want members of my teams to learn and develop. This means that sometimes I’ll be hands off to the point of watching people make safe mistakes. That’s a deliberate choice as I believe people learn a great deal from making mistakes. It’s part of my job to make sure that those mistakes are safe.

4. Enjoyment

We spend a lot of our lives working. We should enjoy it. I try to take a light-hearted approach to everything I do. It’s entirely possible to serious things with a smile on your face and I want my team to be smiling too.

Oh, and I’ll bake cakes too.

5. Temporary relationship

Many companies act as if their employees will be with them forever and if they ever left then that would be a huge breach of trust. I really don’t like this.

I prefer to acknowledge that the employer/employee relationship is a temporary one. Ideally, team members would be with me for at least 2 years. Ideally, those team members will continue to learn and grow whilst they work with me. Ideally, I’d have new roles ready for these people as they grow. The reality is that they will get to a point where the next step is outside of my team and they are moving on to an opportunity I simply cannot provide. That is something to be celebrated. It is an amazing experience to have a team member come to me and say “Colin, I need to resign as I’ve been offered a more senior role that isn’t available here”. when that happens I feel I have done something right.

Conclusion

I know many people have different management styles. I like mine, it makes me happy and leads to effective teams that I’m proud of. I think my management style has a lot of positive feedback loops that make it powerful. Being hands off shows that I trust people which leads to greater trust which leads to more enjoyment.

Startup CTO - Making Salaries Transparent

Tue, 05 Feb 2019 13:00:00 +0000

The fourth part in my series on being a Startup CTO.

I’m a big fan of transparency within my teams. I believe that it shows trust and enables people to make smarter decisions as they have more context to make them.

For me, one of the pinnacles of trust and transparency is enabling people to see the salaries of their co-workers (including their managers).

There are a lot of great posts out there that describe various transparent salary systems. I haven’t seen many that describe the journey to getting these systems rolled out with minimal problems. What follows is how I rolled out a transparent salary scheme in the engineering team at Conversocial.

The Motivation

I had been considering making engineering salaries transparent for some time and kept finding reasons not to do it - being busy, other bigger problems to solve, the usual startup stuff. It all came to a head when one of the engineers, Ed, came to me and said: “I have no evidence for this, but, I feel that I’m underpaid relative to everyone else”. This statement shocked me as I knew that he was one of the best-paid engineers. I wanted to understand why he felt like this, and I asked Ed to draw up a list of everyone else and what he thought their salaries were compared to his in £5k increments.

What he produced was, in hindsight, obvious. Ed was a frugal guy. He didn’t buy fancy clothes, didn’t eat fancy dinners and cycled to work every day. The engineers Ed felt were earning more than him were the ones that lived a more lavish lifestyle. In one case, someone he felt was earning £20k more than him was earning £20k less. The person he was comparing himself to was an accomplished photographer and loved going out to fancy restaurants and taking beautiful photos of the food.

Ed was the final push I needed to move transparent salaries to the top of my list of projects.

Getting buy-in

The CEO

Before jumping in at the deep end, I spent some time talking this over with our CEO. We discussed the pros and cons of going transparent. We were both generally keen to be as transparent as we could with the team, and this felt like a natural next progression. There were some initial concerns:

What if someone gets an offer from elsewhere that is higher than we currently pay?
What if we want to hire someone and we need to pay more to secure them
What if someone is unhappy about someone else’s salary compared to theirs?

We came up with plans for all of these. Most of it was solved by trying to pay fairly and competitively. If we’re struggling to retain or recruit someone then either we’re not competitive (and we need to increase our salaries across the board), or we have to stick to our plan as paying one person the ‘wrong’ amount would cause upset across the rest of the team. This latter point is valid even without a transparent system as people talk about salaries and guess at what other people earn.

The CFO

The conversation with our CFO was very similar to the one with the CEO. Except the CFO wasn’t as motivated by transparency. The CFO was much more concerned with predictability of our engineering budgets and ensuring we were smart with our money. I informed our CFO that the likely outcome of this process was an initial jump in salaries as we level everyone out and a switch to reviewing salaries every 6 months to ensure we’re always competitive.

The initial increase was less of a problem than I imagined. Our CFO understood that the market for engineers was getting more competitive and being able to retain them was important (it helped that I had identified a few engineers as potential flight risks as they had improved significantly and were overdue a raise).

The switch to reviewing salaries every 6 months was probably the biggest hurdle. We eventually agreed that at the start of each year we’d agree a budget for increases for the whole year and I could phase those increases in however I liked. If I then wanted to go beyond those increases we would have to talk to the board and get the budgets changed.

The team

After getting buy-in from our CEO and CFO, I then took the idea to the team at one of our team meetings. I made it clear that I hadn’t yet created a system, but, I wanted to understand how they felt about the idea of creating a more thorough system of setting salaries and that the system would be transparent.

The resounding answer was “this sounds good if you get it right, but, you probably won’t”.

None of the engineers had ever worked in an environment with transparent salary schemes. All of the engineers were used to a system where they asked and campaigned for their pay rises and had no idea if it was fair compared to the other members of the team.

I had must have caught the interest of some of the engineers as I started getting linked to various posts from other companies about how they set up their scheme. Often the engineers would tell me what was good or bad about the schemes.

The system itself

I’m going to skip a detailed discussion of the system itself as it is described in a previous post. It is sufficient to say that the system comprised of an evaluation of approximately 20 different skills and the person’s strength at that skill with 4 different grades of strength for any skill.

Calibrating the system

After devising the rough framework, I sat down and evaluated all the team leads skills and then had them do the same for their engineers. This evaluation allowed me to start spotting patterns and use these patterns to form the various grade boundaries. E.g. an engineer must have most skills demonstrated at a level of ‘self sufficient’ to progress to mid-level.

Getting ready

The calibration process highlighted a few discrepancies. Some people were being paid less than they should. Some people were being paid more than they should be.

For the people that were being paid less, they got a pay rise in advance of the system being made visible to everyone.

For the people that were being paid more I had to come up with a justification. I found a solution in the concept of an “uplift”. The engineers that were being paid more than they should be were being paid these rates for historical reasons with regards to specific skills being hard to find - e.g. javascript engineers that can build complex web applications were scarce at one point. We then created an uplift for these skills and said anyone that demonstrates they can do this will also receive the uplift and as the skill becomes less scarce the uplift will decrease. At the time of my departure from Conversocial all of the uplifts had been removed as people progressed and learnt rarer skills.

For the actual rollout of the scheme, I sat down with each person individually, explained their new grade and why they were at that grade. I explained the system to them and what their salary would be. I then took the time to answer any questions from the engineer.

After this, I then herded everyone into a room and put the spreadsheet of salaries on the wall for everyone to review. I was nervous that despite me being comfortable with the numbers someone would be upset. I planned that if anyone was upset I would attempt to understand why and resolve to fix any unforeseen issues.

What happened? Nothing. It was one of the biggest anti-climaxes I have ever experienced. Everyone looked at it. Everyone saw that they were in the right place and surrounded by people that they felt were peers. Everyone understood the system and its fairness. Everyone was happy.

After the rollout

After going transparent, I had a few interesting things happen.

The highly paid offer

Shortly after implementing a transparent salary scheme at Conversocial I got approached by one of our engineers, Jess. Jess had received an offer from her previous company to come back to them. The salary was approximately 25% more than the salary we were paying. This salary was a huge increase and representative not just of how good Jess is, but, also of specific knowledge that Jess had of the previous company and the specific value she had to them.

Jess explained this offer to me and then instantly said: “I know you can’t change my salary due to the salary scheme, but, I wanted to talk to you about my future here”. Straight away, money was off the table. We then had a productive conversation about how Jess wanted to lead a team and get real experience with leading more junior engineers. I hadn’t realised how strongly Jess wanted to lead a team. I explained to Jess that although there weren’t any opportunities at that specific point in time, the company was growing, my target was to hire another 5 engineers and build another team in the near future and that she’d definitely be up for consideration for a leadership role when the role became available. I was also honest with her. I explained to Jess that I couldn’t promise this role to her even though I felt that she would be great for it. I wanted her to understand that I wouldn’t make a promise I couldn’t 100% keep. I told her that the opportunity she had at her previous company was an excellent salary and not likely to be matched elsewhere. This was a conversation where I laid out facts, was transparent about my goals and my views of her and the company.

Having transparency and honesty in our history meant that Jess trusted me that we’d consider her when the time came. Jess decided to stay with us as she understood the potential for growth with Conversocial was worth more to her than a 25% increase in salary.

Roughly nine months later an opportunity arose, and Jess was made a team lead. Eighteen months further on, Jess was still with Conversocial and a very successful team lead.

I believe that had we not had a strict and transparent salary scheme this conversation would have played out very differently. We would have likely got hung up on the salary and not spent enough time focussing on what Jess really wanted long term. Jess would have likely left for the pay rise.

Critics of these systems argue that I could have just said Jess’s salary was not negotiable until pay review time and still had a productive conversation with her about her career goals. This is partly true. However, Jess would not have known whether I was simply saying that her pay was not negotiable or that it was true. Jess would also have not known whether her pay rise was fair and the rest of the team might eventually have found out. Transparent systems make it obvious to her what the situation is.

Finally, an accomplished negotiator would argue that I could have discussed salary and discussed her future and came to the same end. I would argue that salary is such an emotive subject that once it is on the table it is hard to get away from it.

The keen young engineer

We had a bright young engineer who pushed the salary system at Conversocial. Fred was keen to work his way up the grades at Conversocial.

Every time we did a skills evaluation with Fred would argue that he had demonstrated skill X to the level required and would provide evidence of it. Every time Fred would ask for clarifications about what skill Y on the next grade meant and what he had to do. This was a great conversation to be having. Fred was hungry and was pushing us to help him grow as fast as possible.

As a direct result of this, Fred has grown his skills at an astounding rate, and his grade has improved rapidly in line with this.

Conclusion

Rolling out a transparent salary system need not be hard. Talk to your CEO and your CFO to get their buy-in. Talk to the team to ensure they’re happy with this level of transparency. Take the time to carefully calibrate the system and if all goes well you’ll be greeted with a contented quietness when everyone finds out what their salaries are in relation to everyone else.

Startup CTO - Compensating Engineers

Thu, 24 Jan 2019 13:00:00 +0000

The third part in my series on being a Startup CTO.

A common question I see from startup CTOs is “how do I compensate my engineers?”. This question often arises from a place of wanting to establish some rigour and fairness to the system rather than rewarding someone for being a great negotiator (or great at threatening to leave). Typically, this happens when the engineering team gets above 10 people. Attempting to put a structure to it too soon can be an exercise in spending more effort than the gains justify.

Getting compensation right is essential. In my experience, money isn’t a great motivator for many engineers. However, if done poorly it can be an incredibly powerful demotivator and push people away from your company.

In this post, I talk about how I create a compensation system. This post does not deal with on-call compensation as that is typically a separate concern.

Building a System

Objectives

Whenever I think about designing a compensation system, I like to figure out what the objectives of the system are. For me, this usually boils down to the following:

Transparency - at a minimum, everyone needs to be able to understand why they are getting compensated how they are and what they need to do to get more. In the right cultures, this can be taken to the extreme and salaries of individuals can be made transparent.
Encouraging the right behaviours and development of desired skills
Rigour - the system must enable another engineer to evaluate a coworker and come to a similar answer as another engineer doing the same evaluation
Fairness - engineers should be rewarded for being good at their job and not for singing their praises

A notable absence from this list is making it work for everyone. Whatever system you develop will likely not work for everyone. It is better to be aware of the limitations of your system and ensure you hire to fit them. For example, you might have a very customer-centric culture and reward engineers for being ‘good with people’. This system means that anyone that wants to avoid customer contact will likely not do well in your system. Knowing this enables you to make smarter hiring decisions and keep growing the culture you want.

All of this combined contributes towards a healthy engineering culture with high retention and growth.

What skills do we value?

Most companies loosely agree on what makes a great engineer. There is some variance though and understanding what is important to your organisation will help ensure that you’re rewarding the behaviour you want to see more of. For example, early in Conversocial’s life we needed engineers with a good sense of product and we specifically hired and compensated for this.

Grading systems

Attempting to do a quantitative analysis of an engineer’s performance is hard. Being a great engineer is a combination of soft skills and technical skills - the exact definition of great varies between companies depending on their needs.

I favour using qualitative assessment with a structure to what I’m qualitative around. Typically this results in a grading system where engineers are given a grade according to a variety of qualitative metrics. Building a grading system from scratch is an arduous task and starting with an existing one can be a great shortcut. I often turn to Camille Fournier’s excellent ladder.

Broadly, my guidance on junior vs mid vs senior engineers is as follows:

Senior engineers find problems and fix them. Senior engineers provide guidance and help other engineers grow (via formal coaching or informal methods such as pairing)
Junior engineers typically need helping with every project they work on and would be unable to complete the work without that help
Mid-level engineers can mostly work independently, but, need guidance from more senior engineers from time to time

I don’t mean that senior engineers are off doing their own thing and being mavericks. Senior engineers have the ability to do so - but still seek the opinions of their co-workers to ensure they’re doing the best they can.

As your team grows you will find that you will likely want to evolve the system by adding more grades or different tracks for different skill sets, e.g. a leadership track.

I always make this guidance visible to the team so that they can see what they need to be doing and what we value.

Rare Skill Uplifts

In combination with a grading system, I like to include uplifts in salaries for engineers with particularly desirable technical skills or knowledge. Uplifts are a fixed amount of additional salary. Uplifts help the system cope with market dynamics around particular roles. For example, at the time of writing React engineers are in particular demand and consequently expensive. Making this transparent with the grading system helps engineers understand what technical skills they might want to learn in future.

What should the salaries be?

Giving each grade a single salary (e.g. £50,000) rather than a range keeps things simple. Each grade also has a large gap (~10%) between salaries. This combination keeps decision making easy - you’re not trying to justify why one person gets paid £500 more than someone else. It also makes it easier to evaluate against as you’re looking for step changes and not trying to find small differences in performance.

Don’t be frightened to give individual contributors larger salaries than team leads. They’re different skill sets, and it’s entirely possible for an individual contributor to be more valuable to the company than a junior team lead.

Working hard and bonuses

I typically don’t include raw output in my annual compensation systems. Doing so can lead to people getting rewarded for the wrong things. For example, someone might never help out with code review so that they look like they’re creating lots of great code. Someone else might shy away from the big hairy (but valuable) refactorings in favour of churning out lots of greenfield code. Rewarding an engineer for raw output can lead to burnout as they work harder and harder to try and chase the cash. Instead, I like to try and motivate people to work for the long term and remind them that this is a marathon and not a sprint.

In the rare event that someone has done something beyond the call of duty (e.g. they pulled all the stops out to help deal with an unexpected and unplanned urgent requirement) then I try and do ad-hoc bonuses to say thank you. At the minimum, this is some additional time off to make up for any lost sleep. I try to keep these bonuses as unique non-cash treats of relatively low value (sub £1,000). For example, I had one engineer stick around to help diagnose a particularly tricky deployment that had gone wrong, and we gave him a day off as well as treating him and his partner to a fancy Michelin star dinner on us. I like bonuses to be a unique and enjoyable experience tailored to the individual that they might not otherwise have spent their own money on.

Moving to the new system

After creating a compensation system, you need to move everyone over to it. I like to quietly phase it in for a year to iron out discrepancies and ensure it is working. I use it for evaluations and salary changes, but, not actively talk about the structure with the team.

On day 1 some engineers are underpaid according to the system, and some are overpaid.

The engineers that are underpaid are easy to deal with - increase their salary over the 12 months and give them some great news.

The engineers that are overpaid are trickier. Particularly if it’s a large gap. My first reaction when this happens is to take another look at the compensation system and see if there is a reason this person is getting paid more than they should (perhaps there was a skill that was valued and is no longer required). Overpaid engineers are also the reason for a quiet introduction of the new system. Market salaries are generally increasing each year, and this gives you a chance to give the overpaid engineers slower increases and get them in line with the new system. I have never lowered someone’s salary as this is almost certainly going to lead to resentment and them leaving (and possibly legal implications).

Additional Details

As with any complex system there are a lot of details and questions that arise.

Who does the grading?

Initial grading of an engineer is done during the hiring process by the hiring committee. There is an understanding that if it becomes evident that it’s way off, then the grade is corrected as soon as reasonably possible (within 3 months). Typically we see engineers underperform in interviews and need accelerating through grades rather than overperforming in interviews.

I often turn to team leads to help with grading. I ask team leads to be advocates for the engineers and come with evidence as to why their team members should go up grades.

Frequency of evaluation

I’m a big fan of doing grade evaluations every 6 months. If someone doesn’t quite make a grade jump for whatever reason then waiting 12 months for a bump can be a long time and lead to disgruntled employees. I’m also open to engineers going up grades at every evaluation if they’re genuinely progressing that fast (I most commonly see this with fresh university graduates that are doing lots of additional learning and putting in vast amounts of effort).

Going fully transparent

I usually go fully transparent with my teams. Specifically, every engineer can see the salaries of the other engineers (and my salary too). There is a vast amount of literature out there on why this is great and here are my top reasons:

It keeps you honest and holds you to the system
It gives engineers a chance to find role models and see who to learn from
Engineers sometimes pipe up if they think that someone else is deserving of a grade increase

Doesn’t this tie my hands when making offers to candidates?

I’ve found having a well-structured compensation system has helped me with making offers. Offers are usually framed as

“we have a compensation system to ensure people are getting fairly paid and not being rewarded for shouting the loudest. We have evaluated you as what we would call a Junior 3rd grade, and the salary for that is £X. We realise that we might have got this wrong and we do re-evaluate after 3 months to ensure that people are on the right grade. We also continue to evaluate everyone on the team every 6 months. As well as salary we offer several other great benefits…”

In roughly 95% of offers I’ve made, this has taken money off the table as something to discuss and has moved the offer discussion to more interesting things: who will I get to learn from? What team would I be on? how do I progress?

In the other 5% I’ve encountered a massive mismatch (e.g. we’ve offered £45,000, and they want £65,000). Mismatches usually happen when the candidate has a skill that the market values and we don’t (e.g. they’re a mobile engineer, and we do not need them to be a mobile engineer). I find that discussing salary early in the hiring process alleviates a lot of this.

This is particularly powerful when combined with full transparency as the candidate can be more confident that you will actually do what you say and aren’t doing some underhanded tactics.

Conclusion

Creating a compensation system is hard. Getting it right will make compensation decisions easier and give people guidance on how they should be developing. Often a great compensation system will be mostly painless and result in very little internal discussion around salary. You’ll have more positive conversations around how people can grow. A bad compensation system can result in engineers being unhappy and moving on to other roles.

Startup CTO - Focus

Tue, 11 Dec 2018 10:00:00 +0000

The second part in my series on being a Startup CTO.

It doesn’t matter how fast you go if you’re going in the wrong direction. It’s even worse if everybody is going in different directions.

Focus, in a startup, will make or break the company. I’m not talking about whether engineers can get quality time to focus and write code without distraction. I’m talking about big decisions. Your company’s goal is to solve a particular problem and make a pile of cash doing so. Are you focused on that?

As a startup, you have one resource that is more precious than any other: time. If you waste time on the wrong work then you have no way of getting that back. As an optimisation problem, if you spend 50% of your time on the wrong work then the most significant performance improvement you can do is stop working on the wrong work.

The lure of the sale

One of the largest risks I see to a startup is the lure of the big sale. A big prospect comes along, and they are interested in the problem you are solving…ish. You can do the mental gymnastics to convince yourself that they’re trying to solve the same problem, but, deep down, you know they’re really trying to solve a different problem. They’re close, but, they’re not the same problems. Be honest.

What typically happens in this scenario is that everyone convinces themselves that they are solving the same problem, or, it’s OK to solve this other problem, because, we will eventually convince the customer that our problem is what they should be solving. Following this, the entire product/engineering plan goes down the toilet and 3-6 months are spent trying to satisfy the prospect’s needs.

If you’re lucky/unlucky (I’m not sure which is better yet) the prospect will sign up with you and give you a lot of money. Excellent, all that work was not wasted, and you can get back on to the roadmap… oh, no, no you can’t. They have more needs now that they’re on board. It’s OK, with that extra cash you can hire a few more engineers and another product manager to work for this customer, and you can try selling this stuff to your other customers. Perfect, you’ve ring-fenced the problem, and you can get back to your roadmap having lost 3-6 months (or maybe 9 months now that you’ve spent time hiring this new team).

A month goes by… the customer wants more. They’re your biggest customer, and unsurprisingly, they’re demanding and need more than just 2 or 3 engineers. They’re threatening to cancel. You move more engineers over. This continues for another year or two before the customer leaves. They leave because you weren’t really building what they wanted whilst you tried to juggle two roadmaps.

I’ve played this out slightly more dramatically than it really happens. The killer is that this usually occurs quietly. It happens time and time again at small companies and startups. This dual focus kills companies. With all the time you’ve spent trying to satiate the needs of a prospect that doesn’t want what you’re building you’ve reduced your investment into the problems you wanted to solve in the first place. Your existing customers get disillusioned with you, and they leave. The end.

In most of the discussions I’ve ever seen around this, I see the ringfence argument applied. What is rarely accounted for is the distraction to the senior management team. The time taken by the executives to go and talk to this client and make them feel loved. The time taken by the head of product trying to balance two competing roadmaps. The mental space occupied by having this additional thing floating about when you want to focus on your big problem.

So, how do you avoid it?

Be honest and assess the situation carefully. Gather requirements, gather data and decide as if there was no money on the table. Take the money off the table and see what conclusion you’d reach. In some cases what they’re asking for is what you wanted to build later on, and you’re mostly just changing priorities. Great!

You’ve got to do your homework when a whale of a prospect arrives. It’s easy to get caught up and think you’re the only company they’re talking to. It’s easy to believe that the problems they’re talking about are real problems and they will shower you with money to make them go away. Just remember, talk is cheap. It’s easy for an exec at a company to say “oh we’re having a problem with blah and we think you can help”. Validate their business case. Make sure it stands up.

What do you even want to focus on?

The other significant risk I see in startups is not knowing what problem they want to solve, or, trying to solve the problem with a scattergun approach. They’re not sure what the solution to the problem is, so, they try lots of solutions and see what sticks.

For example, we want to be the market leader for building widgets. We’re not sure if we want to do B2C, B2B enterprise or B2B mid-size. So, let’s build a generic product that appeals to all of these and see what sticks. I can tell you what will stick: nothing. You will not have built a compelling product for any of these markets, and you will have instead wasted the most precious thing you have: time.

It’s OK to make a mistake in choice of focus if you have a way of spotting it quickly and course correcting. For example, you might do your research and decide that B2B enterprise is where your business needs to be. That’s the most natural market for you to disrupt and you have the right skills to do it. Focus on it. Ruthlessly. Everything else is a waste of your time.

Don’t stop validating that you’re focussing on the right thing. Keep on asking the market about what you’re doing. If you get it wrong, re-assess where you need to be and focus on new problems if needed.

Conclusions

I have seen well run, fantastic companies, lose an entire year of engineering time on the wrong things. The company would have been better off sending all the engineers on holiday and closing the office to save money.

Figure out the problem your company exists to solve and focus on it.

Startup CTO - Technology Choices

Sun, 25 Nov 2018 10:00:00 +0000

The first part in my series on being a Startup CTO.

The great thing about being a startup CTO is that you’re free to make your own technology choices. The down side is that if you make a poor technology choice you will be stuck with it and likely unable to undo the decision for some time (hopefully you’re too busy building your product). I will admit that I have made mistakes with this in the past and I hope to help you avoid this.

I’ve seen many startups do what I call “post-decision-justification”. They make a decision and then justify it the other way around. For example, they decide to write the product in Go because the lead engineer has been learning it and thinks it is cool. They won’t say that though, instead, they’ll argue that the language offers them faster compile times, easier to write code, better type safety, etc. In truth though, someone has an itch to learn and try something new and they’re about to inflict it on your fledgling startup.

A startup is not the place to be trying out new technologies without very good reason. Your business should be moving fast enough that the time taken to learn new technologies and discover their rough edges could be a problem. Often this is countered with “but, we save so much time because we’re using X”. It’s true, some languages are faster for writing code in. It’s quicker for me to write a short program in Python than it is in C or Java. What is rarely factored in is the time taken to understand an unforeseen issue as you scale with the language. For example, do you know how to use the profiling tools with the language you’re about to choose? Do you know how to optimise memory usage in this language?

It’s not just programming languages. We have the same issues with frameworks and databases. Do you know how to perform a zero downtime upgrade with the database that you’re about to choose? Do you know how to properly do backups and restores that work 99.999% of the time? How easy is it do debug a 50,000 line app written using that new framework?

Taken to the extreme I see startups that allow their engineers to use any tool fit for the job. As a result, they have tens or hundreds of microservices all written in a different language using a different framework. You will be maintaining and deploying these services for a long time after they were initially written. It’s probable that the engineer that wrote a particular microservice in Haskell will move eventually leave your company.

How I make Technology choices

Whenever performing technology selection I (with my team) attempt to lay out on the table everything that might be relevant and also flag up anything that I think is cool. I don’t want to sweep the cool factor under the rug, it needs acknowledging so that you can realise if it is impacting your decision.

From there I try to reach an understanding of the full cost of that choice.

How experienced are we with the technology?
Will we be able to maintain it?
What’s the licensing?
Are there debugging and profiling tools available? Do we know them?
How big and active is the community?
Does this impact hiring?
What are the support options? Are there commercial outfits that can help?
How easy is it to recruit people that already know it?
Is this technology giving us something uniquely special?

The last point is particularly interesting. I believe this is the most important reason to try out a new unfamiliar technology. If you believe that the technology will give you something unique that none of the other technologies will then it might be worth using. For example, in the early days of Conversocial (circa 2011) we made the decision to switch from MySQL to MongoDB. At the time the social networks that we were interfacing with were making regular large changes to their APIs and consequently our data models often had to change. As a result of this we were having to take Conversocial offline to make schema alterations and the downtimes were only going to get longer as our database grew. Unfortunately, none of the standard database products at the time offered online table alterations. We investigated MongoDB and decided that it would give us this functionality whilst also satisfying many of our other criteria around maintainability. This decision was not taken lightly and we hit many bumps on the way due to our lack of familiarity with MongoDB. Indeed, it was such a new product that some of the issues we hit were new issues that nobody else in the world had hit previously. This is the kind of cost that is rarely factored in when making technology choices. Overall, I think it was the right decision at the time - we gained our needed flexibility and didn’t have to compromise too much on our other needs (we were fortunate in not needing transactions or being able to workaround the lack of them).

Since then, technology has moved on. If I were to be making this same decision now, I would not choose MongoDB as both Postgres and MySQL now offer online alteration and have far strong communities as well most people being familiar with them.

This doesn’t sound innovative

Correct. For most startups I believe that product innovation is incredibly important. Technology innovation can be a distraction that doesn’t actually help build the business or the product.

Introducing new technology

If you do decide that you want to introduce new technologies then there are a few things you can do to ease the way.

Isolate at first

If possible, use the new technology in as isolated a way as possible. Use it for some small part of your system and let it bed-in before widening its use. My general rule of thumb is to not widen the use of a new technology until:

it has been in production without issue for three months
you’ve done an upgrade of it
you’ve had to debug a horrible issue with it - I often find this is where you find that something is not as shiny as you thought

Once you’ve done those things you’ve probably uncovered most of the warts and can decide whether further rollout is actually a good idea or not.

Hack days

You need to try and get people up to speed with the new technology and discovering issues as quickly as possible. It helps to have the new technologies embraced and known by as many people in your teams as possible. For this reason, I would often host hack days prior to introducing a new technology. For example, I had an internal hack day where everyone learnt React and build something with it, I did the same with Solr and ES6. In all these cases I would have someone learn as much as they could about these technologies prior to the day and act as a guide on the day.

Out with the old

Once you decide to introduce a new technology it is worth considering if an old technology can be removed from your system. If you continually add new technologies eventually your stack becomes difficult to learn. An engineer joining you needs to learn five different database technologies and three programming languages before they can really get productive for you.

Conclusions

New technology can have a place in every startup. Before chasing the shiny new, think carefully if you would be truly better off with the technology you know. If you still want the shiny, introduce it deliberately and with care.

The Startup CTO book will be blogged

Sun, 25 Nov 2018 10:00:00 +0000

So, I wrote a book on being a CTO in a startup and everything that I learnt the hard way. Unfortunately, I never got it to a place where I was happy with the structure and so I never published it.

When I first started in the CTO role I found myself needing to have knowledge of a broad range of subjects, some of which I didn’t even know existed. I often found myself having to quickly research specific topics or think on my feet. This wasn’t ideal.

My intention with the book was to give someone starting out in a startup CTO role a single place to go to give them a brief grounding in everything they might need to know. I still believe this is a worthwhile aim.

Having done a bit of soul searching I have decided to rewrite each chapter as an individual blog post and make it available for free. Enjoy :)

Technical interview exercises

Wed, 30 May 2018 11:00:00 +0100

I’ve interviewed a fair number of software engineers, hired a bunch and had to say goodbye to a few due to my own mistakes in hiring. As a result of this I have landed on an interview setup that works for me, my teams and (perhaps most importantly) the candidate.

This is a post about how I run the technical interview after a phone screen has established that the candidate is looking like a potential fit for us. I will likely follow this post with another looking at cultural fit, growth potential etc.

It’s an engineers market

One thing to bear in mind when hiring engineers is that it is generally an engineer’s market. There are more good jobs than there are good engineers. You’re in competition with a lot of other companies and so you have to try and make this process as rewarding as possible for the candidate to ensure that they stay interested in your company.

Even if it was not an engineer’s market it is always worth making the process decent for the candidate. They’re another human being and interviews can be some of the most stressful times for someone.

Why we interview

Before getting into this it’s worth taking the time to think about why we interview. There are a few things we’re doing:

Understanding if the role/company/candidate are a fit for each other? It’s a two-way thing
Selling the company/role to the candidate - the candidate needs to come away with a good impression. Candidates will talk to their peers, so, it’s important that the candidate has a good impression even if they don’t get the role
Attempting to minimise the number of mis-hires and number of missed opportunities.
Learning and having fun - if you’re bored the candidate will soon realise and leave with a bad impression

In my experience a lot of energy is spent on thinking about company/candidate fit and not as much energy is spent on thinking about role/candidate fit and everyone does the same standard exercises that lead to mis-hires.

Role/candidate fit

Let’s take a look at what a typical software engineer might need to do in a role:

Coding (high-level / low-level)
Testing
Deploying
Debugging
Technical design
Architectural design
UX/visual design
Documentation
Talk to customers
Performance optimisation
System administration / devops
Mentoring
And the list goes on…

It’s a long list and you can’t interview for all of these skills, otherwise, your interviews will be either very shallow or very long. On top of this, if you spend a lot of time interviewing for skills that aren’t used very often in the role (e.g. performance optimisation) then you’ll be giving a false impression of the role to the candidate.

Pick a few skills that matter (3 or 4) and interview for those in as realistic a way as possible - setup 30 minute exercises that actually demonstrate the skill. This not only helps you establish skill levels, but, allows the candidate to see what kind of work they might be doing.

The most important skill that I haven’t mentioned is learning as this runs through all of the other skills. Throughout the interview I am looking for people that can learn quickly.

As well as learning, the top four skills I focus on are: architectural design, testing, coding and debugging. These are the bread and butter for what most engineers I hire will be doing.

The exercises

Introduction

All of my interviews start with an introduction. All of the interviewers introduce themselves, a bit of their background and their role in the company. We then ask the candidate to do the same.

We then explain how the process works, what the exercises are and what we’re looking for. I always explain the following:

the interview is hard and nobody does perfectly
the interview is tight on time so we may cut some exercises short if we feel we’ve learnt everything we need to know - typically this is because we’ve been enjoying talking to the candidate or there’s been a point of confusion
we’re not looking for the right answers, we’re looking for the boundaries of their knowledge and how they learn when they hit these boundaries

We then reverse the standard process and ask if the candidate has any questions at all and that they should always feel free to ask questions throughout the interview. We won’t judge any questions, but, we may refuse to answer the question if it’s going to obviously give them the answer.

My architectural design exercise

Architecture is a very broad topic. As a result I have tailored this exercise to the startup environment that I operate in.

Typically, this exercise is a whiteboarding session (paper is also offered if the candidate prefers). The interviewers then say something like:

We’re a group of entrepreneurs and we’ve come to you as a technical expert to build us a system that allows users to do X.

The X is normally some kind of system that allows people to interact with each other in some form. For starters we’re looking for people that ask questions and attempt to gather requirements in some methodical manner. From here, candidates typically fall into two groups: breadth first or depth first. Both are fine.

We’re looking for candidates that build a simple system first and then we start throwing spanners in the works. What happens if we need to cope with X thousand concurrent users? What if the database blows up? What about hackers?

The best candidates will generally build a simple system and adapt it as we throw in more requirements. They will explain their decision making process (or we ask them and they can explain). The best candidates will allow for decisions to be made for human reasons - e.g. I would probably write this all in Python as that is what the team knows and it’s only a simple web app.

The testing exercise

This is another whiteboard exercise:

You work at NASA and you’ve received a program. The program runs on the terminal and does Y. As this is NASA we really need to know that Y works and won’t blow up our space ships. You don’t know what language it was written in, you just have the binary. What kind of test cases do you think you need?

Y is normally a really simple task like counting words in a string or something. We typically then write up a sample test case to get them going.

The best candidates will do a few of the ‘happy’ cases and then start thinking about edge cases. Their prior experience will often shape what kind of edge cases they think of (e.g. people with a history of C will start trying to make buffers overflow). This is normally a great chance to throw in some curve balls and get a chance to demonstrate learning. E.g. you might start talking about floating point precision and how that might break the program.

The coding exercise

The coding exercise is normally done as a test-driven development ping-pong exercise. The task is to build the program that was being tested in the previous exercise.

The best candidates will get into a good flow of writing simple code, getting tests passing and then refactoring. The best candidates will explain what they’re trying to do and write good clear code.

The debugging exercise

I was nervous when I first added this exercise to my standard set. It’s a bit naughty. I normally write a simple program that has no bugs in it and then I tinker with some underlying library, or OS configuration (for devops roles), and introduce a bug (e.g. I tweaked the MySQL driver to only ever return odd numbers when doing a count). This exercise is introduced as being a simple program that has been written and then a bug has been created somewhere in the system.

The program being debugged is intentionally very simple and looks something like the following pseudocode:

db = get_connection()
db.table.delete_all()
db.table.insert([
    'bob has %d friends' % (i * 2)
    for i in range(200)
])
print('My program inserted %d records' % db.table.count())

This is a heavily coached exercise and it generally only takes 20-30 minutes for someone to figure out what is wrong (90% of people get to the right answer). We make it very clear that we’re more interested in how they approach this particular problem than whether they actually get to the root cause or not.

The full (modified) source code for everything is available to the candidate, as are debugging tools. We also encourage modification of the code and use of other utilities to verify things if that is what they want to do.

The best candidates will do a binary search to establish where in the program the bug is happening. They will then come up with some hypotheses and attempt to prove them. They will try and verify behaviour using other tools (e.g. use the standard MySQL client). The best candidates will look up documentation, but, be open to the documentation being incorrect. They won’t be frightened to dive into the source code for lower levels or open a debugger.

Informal chat

After the main exercises are complete we get someone from outside the process to have an informal chat with the candidate for 10 minutes whilst the interviewers answer the question: do we need to know more or can we make a decision? We’re not trying to make the decision here, just establish if we know what we need to.

Wrap up

We then give the candidate another chance to ask questions before letting them know we’ll be in touch in the next day or two with feedback.

Post-interview

After the interview each interviewer then immediately scores each exercise out of 5, writes up any good/bad things they noticed and then decides their overall verdict. Once this is done the interviewers get together and do a simple thumbs exercise. After a count of 3, give either two thumbs up, 1 thumb up, 1 thumb down or 2 thumbs down. We record this result for future analysis. Then the hire/don’t hire debate begins!

It’s important that all of the scoring and decision making is done independently to avoid anyone influencing anyone else.

Once a decision is made, feedback is given as quickly as possible (typically the same day or next day). Giving high quality feedback quickly will give the candidate a really good impression of you - even if the answer is a no. In the past I have rejected people for a role, but, have received indirect feedback that they really enjoyed the interview and learnt a lot. This is the result I’m aiming for.

Conclusion

I think I’ve landed on a pretty good process. It has reduced the number of mis-hires I’ve made and has also enabled me to make a few hires I might not have made otherwise (people that I know would have done poorly in interviews that some other companies run).

I’d love to hear thoughts and feedback on this. It’s definitely something that I continue to try and improve!

Tech Debt and Impossible Mountains

Thu, 03 May 2018 11:00:00 +0100

I’ve been thinking a lot about tech debt lately. I’ve always felt like there are two kinds of tech debt: the sort that you can comfortably ignore, and the kind that slowly strangles you.

Having built and maintained a few systems, I’ve now crystalised my thinking. All technical debt starts life as a small pile. Over time you might build on top of that pile and it eventually becomes a hill. It might even become a mountain. That’s OK though. We can slowly climb a mountain if we can break it down into manageable steps and take a breath on the way. We can slowly fix this kind of technical debt.

Sometimes though, we build impossible mountains. Mountains that you can’t break into small segments as you’ll just fall down. The kind of tech debt that has to be dealt with in one go and it’s built up enough that it becomes infeasible to deal with - the business simply can’t afford you to down tools for six months whilst you resolve it.

Some Examples

Below are a few real world examples that I’ve dealt with.

Impossible Mountain 1 - The Slow Test Suite

I love tests. You should too. Given enough time and not enough attention, test suites may eventually become slow and you lose the ability to rapidly execute them as you are building software. This is often an impossible mountain. Unless you’re fortunate you’re unlikely to be able to find a small change that will reduce a test suite run time from twenty minutes down to two. Instead, you have to break it down into a series of small changes going from twenty minutes down to nineteen, down to eighteen… all while the rest of the team are continuing to build more tests. At this point you find it hard to gain traction to take the time to fix the problem.

Fortunately, you can build an escape valve to prevent this kind of problem. Most people will agree that a fast test suite is valuable. Most people will also be happy to agree that if the suite goes over a certain time threshold then it needs to be dealt with before it becomes too big a problem and the value of the test suite diminishes.

Impossible Mountain 2 - Leaky Abstractions

This pattern is more sinister. Leaky abstractions make refactoring hard. My favourite example is using an ORM. Often we use an ORM to make dealing with a database easy and safe. We write code that looks like this:

def send_daily_emails():
  for user in User.objects.filter(notifications_enabled=True):
      # code to generate the email goes here
      send_email(user.email, report)

At first blush, this seems fine. Then we decide that, actually, we need to refactor the User model. Supposed now, notifications should be a series of different flags, or, the User objects are being split into two different types. At this point we discover we have hundreds of slightly different ways of querying the User model. Now, it’s hard to refactor. How do we check that all the uses of the ORM are going to be OK? How do we check that all the queries are going to be performant when we change our indexes? Now, it might just be another impossible mountain to climb and instead of tidying things… you add another type of query.

Fortunately, this can be avoided with a bit of forward planning. Building narrow APIs in front of your leaky abstractions (in this case the ORM) can make it vastly easier to refactor and analyse what is going on:

def send_daily_emails():
  for user in UserRepo.find_all_with_notifications_enabled():
      # code to generate the email goes here
      send_email(user.email, report)

When we next come to make a change to the User model we can easily find everything that uses that method and change it… if needed. It’s likely you will be able to just change the internals of that method and never change anything that calls it.

Identifying The Impossible

It’s hard to spot the impossible mountains as they’re getting built. I rely on a few smells:

Am I building on top of this issue?
Will changing this be feasible in a piecemeal fashion that delivers value?
Have I heard recurring complaints about this?

These are the bits of tech debt that I focus my energy on and push for time to solve / prevent. The rest, I can live with in the interest of moving the product forward.

Coming back online

Sun, 17 Dec 2017 10:00:00 +0000

It has been too long. I last wrote on my blog in 2012. This was not long after I started at Conversocial. Since then it has been a bit of a rollercoaster. I spent around six a half years helping build Conversocial from a team of 2ish (it’s complicated) up to 80 full time employees. It was a busy time and I deprioritised my writing.

At the end of 2016 I decided it was time for me to move on from Conversocial. It was time for something new, but, I did not know what. So, I took some time out and did a few things:

I relaxed and unwound - being a CTO in a startup is taxing!
I spent a lot of time baking
I caught up on a load of tech that I had been meaning to learn
I wrote the first draft of a book that contains everything I wish I’d known when I first started at Conversocial. At some point I will publish this in some form

Having spent some time writing I have remembered how much I enjoyed it and also how much it helps me. So, I’m baaaack! :)

I’ve decided to overhaul my blog. This means my old entries are gone. I considered converting them all, but, they’re five years out of date and I think that the technical aspects of them might not be entirely reliable now. I’ll keep an eye on the analytics and see if any need resurrecting.

What I am up to now

I’ve recently (6 months ago) started at another London based startup, Juggle Jobs. I’ve got back into tiny things! We’re currently 5ish people (we have 5 full time and a few people that help us out from time to time). It is in an area I never thought I would ever be in: recruitment. I have a distaste for the recruitment industry. As a sweeping generalisation it is full of agents that act entirely in their own self interest and care little for either the people they are representing or the businesses they are working with. I met the founder of Juggle Jobs, Romanie Thomas. Romanie convinced me that it’s time we changed the industry.

We’re now in the process of building a platform that makes recruitment better. We have a particular focus on enabling the future of work. By this, I mean anything that isn’t the traditional 9 to 5 in an office 5 days a week. Flexible working, part-time working, remote working, unusual hours. All of it.

I’m excited about enabling people to work in ways that really suit their lives. Let’s stop compromising our happiness for our work.