The first part in my series on being a Startup CTO.
The great thing about being a startup CTO is that you’re free to make your own technology choices. The down side is that if you make a poor technology choice you will be stuck with it and likely unable to undo the decision for some time (hopefully you’re too busy building your product). I will admit that I have made mistakes with this in the past and I hope to help you avoid this.
I’ve seen many startups do what I call “post-decision-justification”. They make a decision and then justify it the other way around. For example, they decide to write the product in Go because the lead engineer has been learning it and thinks it is cool. They won’t say that though, instead, they’ll argue that the language offers them faster compile times, easier to write code, better type safety, etc. In truth though, someone has an itch to learn and try something new and they’re about to inflict it on your fledgling startup.
A startup is not the place to be trying out new technologies without very good reason. Your business should be moving fast enough that the time taken to learn new technologies and discover their rough edges could be a problem. Often this is countered with “but, we save so much time because we’re using X”. It’s true, some languages are faster for writing code in. It’s quicker for me to write a short program in Python than it is in C or Java. What is rarely factored in is the time taken to understand an unforeseen issue as you scale with the language. For example, do you know how to use the profiling tools with the language you’re about to choose? Do you know how to optimise memory usage in this language?
It’s not just programming languages. We have the same issues with frameworks and databases. Do you know how to perform a zero downtime upgrade with the database that you’re about to choose? Do you know how to properly do backups and restores that work 99.999% of the time? How easy is it do debug a 50,000 line app written using that new framework?
Taken to the extreme I see startups that allow their engineers to use any tool fit for the job. As a result, they have tens or hundreds of microservices all written in a different language using a different framework. You will be maintaining and deploying these services for a long time after they were initially written. It’s probable that the engineer that wrote a particular microservice in Haskell will move eventually leave your company.
Whenever performing technology selection I (with my team) attempt to lay out on the table everything that might be relevant and also flag up anything that I think is cool. I don’t want to sweep the cool factor under the rug, it needs acknowledging so that you can realise if it is impacting your decision.
From there I try to reach an understanding of the full cost of that choice.
The last point is particularly interesting. I believe this is the most important reason to try out a new unfamiliar technology. If you believe that the technology will give you something unique that none of the other technologies will then it might be worth using. For example, in the early days of Conversocial (circa 2011) we made the decision to switch from MySQL to MongoDB. At the time the social networks that we were interfacing with were making regular large changes to their APIs and consequently our data models often had to change. As a result of this we were having to take Conversocial offline to make schema alterations and the downtimes were only going to get longer as our database grew. Unfortunately, none of the standard database products at the time offered online table alterations. We investigated MongoDB and decided that it would give us this functionality whilst also satisfying many of our other criteria around maintainability. This decision was not taken lightly and we hit many bumps on the way due to our lack of familiarity with MongoDB. Indeed, it was such a new product that some of the issues we hit were new issues that nobody else in the world had hit previously. This is the kind of cost that is rarely factored in when making technology choices. Overall, I think it was the right decision at the time - we gained our needed flexibility and didn’t have to compromise too much on our other needs (we were fortunate in not needing transactions or being able to workaround the lack of them).
Since then, technology has moved on. If I were to be making this same decision now, I would not choose MongoDB as both Postgres and MySQL now offer online alteration and have far strong communities as well most people being familiar with them.
Correct. For most startups I believe that product innovation is incredibly important. Technology innovation can be a distraction that doesn’t actually help build the business or the product.
If you do decide that you want to introduce new technologies then there are a few things you can do to ease the way.
If possible, use the new technology in as isolated a way as possible. Use it for some small part of your system and let it bed-in before widening its use. My general rule of thumb is to not widen the use of a new technology until:
Once you’ve done those things you’ve probably uncovered most of the warts and can decide whether further rollout is actually a good idea or not.
You need to try and get people up to speed with the new technology and discovering issues as quickly as possible. It helps to have the new technologies embraced and known by as many people in your teams as possible. For this reason, I would often host hack days prior to introducing a new technology. For example, I had an internal hack day where everyone learnt React and build something with it, I did the same with Solr and ES6. In all these cases I would have someone learn as much as they could about these technologies prior to the day and act as a guide on the day.
Once you decide to introduce a new technology it is worth considering if an old technology can be removed from your system. If you continually add new technologies eventually your stack becomes difficult to learn. An engineer joining you needs to learn five different database technologies and three programming languages before they can really get productive for you.
New technology can have a place in every startup. Before chasing the shiny new, think carefully if you would be truly better off with the technology you know. If you still want the shiny, introduce it deliberately and with care.