Learnings from 80 startup CTOs
This article was originally posted in Medium.
Some days ago I participated in a startup CTO meet-up organized by Rodrigo Martinez, an early stage investor at Berlin-based VC firm Point Nine Capital. There were the top Spanish startup CTOs and several other European CTOs. I have summarized the notes I took.
Hiring
One of the biggest challenges of the tech ecosystem right now is hiring great talent. The talent war in Spain isn’t yet comparable to what’s happening in Silicon Valley, New York or London; but it’s getting worse, specially in Barcelona. Virtually all CTOs complained about it. There were even many friendly gibes among various CTOs for “stealing” talent.
Luis Peralta, ex-CTO of Tuenti, shared with us the questions every startup should ask themselves before hiring:
- What are your values? Values are the starting point for having an aligned team.
- What are the behaviors and skills you want to reward? Technical excellence, team work, honesty, communication, design?
- What are the behaviors you will not tolerate? Are you willing to hire really good people even if they are jerks?
It’s important to create a hiring brand. People want to be in winning teams and working with others they respect. You should promote the technical work you are doing by writing technical articles, open-sourcing libraries and talking at technical meet-ups. Tuenti was famous for its hard hiring process which attracted talented engineers because they wanted to test themselves even if they weren’t attracted to the company at first.
An important point for hiring is the timing. Startups usually hire after receiving investment and they need to do it quickly to implement the plans they have sold to investors. As Samu Fuentes, CTO of Delivery Hero, shared with us, it’s smart to start interviewing 1–2 months before closing the round in order to find the best candidates. I did so in Selltag. We started interviewing people one month before the money was wired, so we could hire two great engineers, a marketing person and an intern in the following two weeks.
Be careful also with oversizing the team. Most problems in organizations come from communication or interpersonal problems. I believe lots of startups hire too fast, losing focus and therefore doing a poor job prioritizing. I’ve seen many examples of small-medium startups with 2–4 iOS developers and another 2–4 Android developers. Few of them really needed so many mobile developers.
Another topic of discussion was the danger of hiring someone whose main priority is the salary, because he/she may jump to another company once he/she has something to show off. You need committed people that believe on the mission of the company. It’s not something easy to discover, but you can sense it.
Employee On-boarding
Congrats, you have convinced someone to join your startup, but how long until that new employee reaches full productivity? That’s the most important thing you have to do, he/she has to integrate with the culture, learn the new technologies and be part of the team. Tuenti and CartoDB have done great jobs at this. In fact, they believe Day 1 is key for the employee satisfaction. Some companies believe a developer has to commit code to production on the first week or even the first day!
A good idea is to have a mentor figure for the first months. You should have an employee handbook with all the usual questions (written by HR). You should have all processes documented. And for developers it’s a really good idea to have a development environment with Docker so he/she doesn’t have to spend several days configuring libraries. Another great idea is to organize lunches between two employees so there are more connection between different teams.
Management
This is the main job of the CTO. As your team grows it will become harder and harder.
Team structure
Many companies shared their structures on the event. There are several divisions you can do:
- Core Team vs Solutions Team. Javi Santana (CTO of CartoDB) told us that their Core Team builds the CartoDB core and their Solutions Team makes the integration with its clients. The key thing is to communicate well between teams so the Core Team understands the problems that the Solutions Team detect talking to the clients. Also, it’s important that people in the Solutions Team don’t feel like they are doing a lesser job than the Core Team. Both are really important.
- Backlog Team vs Long term Team. Jordi Romero (Redbooth) shared this approach with us. Many companies keep focusing on the infinite backlog and they stop thinking ahead, with solutions that can improve the company 5–10x. Redbooth has a team with full-stack engineers to test ideas quickly. Those engineers must feel comfortable doing dirty code and throwing out code. (Side note: I believe it’s dangerous to focus too much on your backlog. It will soon be inundated by nice-to-add features, small bugs and features you thought 6 months ago. You should clean and re-do the backlog every month)
- Backlog Team vs Short term Team. I have heard this idea from 8fit, that was testing this. The idea is just the opposite of the previous structure. Lots of times the backlog constrains you too much, any growth idea is added to the backlog and it will end up implemented several weeks later. The idea is to have a small team that test many ideas on production each week, specially on growth. If they work, they will be redone by the main team. Many of the companies there admitted that they do this informally (even without other members of their team knowing).
When you have different platforms (desktop, mobile apps, TV) you can also structure your team on different ways:
- Different teams per platform. This is the usual: a different team for web, iOS, Android, TVs… They are really specialized teams. As a pro, they have really good knowledge and it’s usually what engineers want. As cons, they could feel lost on what other platforms are doing and they need more communication between teams.
- Different teams per feature (cross-platform). You mix a mobile dev, a designer, a back-end and a front-end. As a pro, they will communicate better doing that feature. As a con, you aren’t always with big projects so it’s complicated to do this if you have small tasks that only requiere 1–2 days.
Within each team, Tuenti has a Product Manager, a Tech Lead and 3 to 8 engineers. The Tech Lead reports to the Lead Engineer, and the Product Manager reports to the VP of Product. All responsibilities are shared between both of them, deploy included. That way they work together instead of pointing fingers. JobAndTalent has a similar structure (many of their employees come from Tuenti). On Scytl each team can decide its own methodology (Scrum, Kanban…).
Communication
The key to a great company is great communication. It usually is the biggest problem for startups that are growing really fast. I believe it’s difficult to over communicate, so be sure everyone is on the same path.
Sergio Espeja (CTO of JobAndTalent) shared with us that they have had many problems on this sense, making them going slower than desired. What they did was to organize teams so they could be 100% responsible of each feature, doing pull-requests of core libraries if needed. That way they reduced the needed communication between teams as each of them was independent.
Luis Peralta (Tuenti) shared with us the Cola Cao Principle.
Cola Cao is a high-energy chocolate drink from Spain. To prepare it you mix milk with Cola Cao. A key aspect is that it doesn’t dissolve quickly. So you should add a spoon of Cola Cao, then some milk, then another spoon, then more milk…
So let’s imagine that the team is the milk, and the Cola Cao is the culture and processes. If you add too much milk (hire many people quickly) and then you add Cola Cao (try to create your own culture and efficient processes), it won’t mix up. You need to hire some people, start creating your culture and processes, then hire more people. That’s how you will build a cohesive team.
Define when a task is done
There was unanimity on this topic. A task is done when it has been tested and deployed to production. If you haven’t deployed it to production, it isn’t done. This idea is something that many engineers do not like. On Redbooth they had to create the expression “done-done” so people understand it isn’t enough to develop it, the PM must approve it, the QA has to validate it and it has to be deployed.
This is a debate I already had in Touristeye and we also discussed in Selltag. At the end we agreed on that. In our Trello board, a task card stayed in the current sprint board on the “QA” column until it was tested by someone and on the “Ready for deploy” column until it was deployed. That way we forced ourselves to test and deploy continuously because we didn’t allowed ourselves to have too many cards on each column. When we accumulated many iOS/Android task cards, it was a sign that we were trying to do too much before releasing the update.
Post-mortem
Aside from doing retrospectives after every other sprint, you should do a special post-mortem when something major goes wrong. For example: when a deploy has broken the web, when the payment system breaks, when you release an app update with bugs, etc. Do not blame, it is important that employees do not think it’s a witch hunt. The important thing is to seek improvements in the process. If you have uploaded a version with an important bug, the QA was not enough.
Refactors and 20% of their time
Developers love to refactor and improve their code. As they learn, they see better ways to implement what they did months ago or what others previously did. It’s a very positive quality, but we have to combine it with the ability to iterate quickly, because that’s the main need of a startup.
How to organize these refactors plus small projects (20% of the time Google-style)?
- One day a week. The pro is that it happens more often. The cons are that context switching is difficult and one day is not enough so you end up doing too little.
- One week each month. The pro is that they can focus. The downside is that the CEO/PMs may think you’re wasting the time, especially if you need to iterate fast.
- Payload model. Instead of planning it, refactor every time you go through that code again. Xavier Beumala told us that in Marfeel they have this mantra “If you see an error, fix it; don’t add it to the backlog”. As a pro, the team has a dynamic of continuous improvement and they may be more attentive to find bugs. As a con, you may think you can fix or refactor it on one hour and you end up spending the whole day. Sometimes it’s better to plan it.
- Refactors as another source of tasks for the next sprint. That’s what we used in Selltag, copying the idea from what they do in Cabify. We have an Engineering Trello board where developers add what they want to improve. On each Sprint meeting, apart from adding tasks from the backlog they add some from this board.
Probably each method is valid for a different stage of the startup. It depends on your needs (iterate to find product-market fit vs consolidate what you have) and the size of your team.
Removing functionality
It was discussed in the context of SaaS companies. When do you decide to remove a feature that few customers use? One of the startups commented, for example, that they removed a feature that practically nobody used but several days a major customer complained. There was just one person in that organization who used it and it was the main reason they used the tool.
CartoDB, as they open-source most of its code, can operate slightly different. If they decide to remove a feature, they keep it for 2 years on their platform. After that time if the client still wants it, they can install the open source code.
Purchasing and integrating companies
Joan Villalta (Scytl) explained to us that from 2011 to 2014 they bought 6 companies in several countries, each one with different technologies. Now they are 250 people in the product team on 7 cities (Barcelona, Athens, Toronto, Oklahoma, Tampa, Lima and Brasilia). One of the great challenges they still have is the communication between offices in order to avoid silos and to be sure everyone has clear objectives. Another problem comes when you have a question and you have to wait for another person in another time zone to answer. So you end up solving issues within 48 hours instead of 5 minutes. They are changing the structure to have more independent teams to try to avoid this.
Samu Fuentes (Delivery Hero) also shared a similar experience. His company was in a hot sector, they used part of the investment to buy local competitors in different countries. They couldn’t plan the integrations nor merge the platforms (different technologies) as they had to focus on growth. Now, they are planning on unifying the platforms.
Rakuten has also faced this problem. Their solution has been to replace certain blocks of the companies they buy, instead of trying to merge the platforms which are similar. For example, the user and payment systems.
Having different offices
Apart from the previous examples, Tuenti also had problems with its office in Barcelona (their main one is in Madrid). They allowed the people in Barcelona to do things differently and it ended up being a problem because it caused separation across teams. I wonder what suggestions can give us startups with different offices which work on the same product.
Budgets
As the startup grows you end up with more expenses and more services. A good idea is to do what John Carvajal (CTO of Kantox) proposes. If you aren’t sure if a service you are paying is really needed, you tell your team you are going to kill it and you wait until you see how many people yell. If nobody does, you can stop paying. For example, he switched off the staging severs on a Friday, and nobody complained until the following Thursday. There you have a way of saving money.
Budgets are also a complicated topic because it’s related to the sense of power in the company. I really liked this post of Ben Horowitz “How to Ruin Your Company with One Bad Process”
The objective was for each manager to build the largest organization possible and thereby expand the importance of his function. Through the transitive property of status, he could increase his own importance as well. Now you may be thinking, “That wouldn’t happen in my company. Most of my staff would never play that game.” Well, that’s the beauty of the game. It only takes one player to opt in, because once someone starts playing, everybody is going in — and they are going in hard.
Personal Development
On Tuenti they have clear processes about this. First, they have documented the responsibilities of each job, when it’s expected to get promoted, the differences between roles. That way it doesn’t depend on each boss, it’s a company policy so it doesn’t create frustrations on employees.
They have weekly one-one’s with each employee and bi-weekly for each senior engineer. There have quarterly 360 performance evaluations on which responsibilities are mapped into question sets. They are testing Worklytics.
A key aspect is that in Tuenti, developers can choose if they want to go into management or being a principle engineer. Both have the same range of salary but different abilities and responsibilities. That way they break the tradicional idea that if you want to earn more and progress in your work life you must go into management.
Social life
Luis Peralta (Tuenti) shared with us his belief that one of the keys of their success was having a team who, apart from working together, also went to parties together or drank beers after work. They were a united team. And probably this attitude was fostered by non-Spaniard employees who wanted to discover Madrid.
Someone asked how do you combine this if you have employees on their 20s with employees on their 30s with children. You can’t expect parents to go to after work drinks or parties. Can’t it create a barrier? There was no clear answer, aside from not enforcing it from the company but trying to be natural about it.
Project Management Tools
This was a topic that was barely touched. At the end, it doesn’t matter if you use Jira, Github, Redbooth, Trello or others. The problem is always communication between employees and making sure priorities and responsibilities are clear. There are startups as CartoDB which have all the tasks on Github, including its roadmap (they have an open mentality). And there are others that use Jira despite its complexity.
Tech
Mobile — QA
Probably the main problem we all face in the mobile world is how to test our apps with enough confidence that they will behave correctly in the real world. QA isn’t easy, and it’s a discipline denigrated by developers. They don’t want to do QA, so you end up with juniors or with an externalized company. A good idea is to have several levels of QA tests, and depending on the changes, the QA team will test one checklist or more. But you always have to check the main actions of your app (signup, create action, search, pay).
Several of them agreed that you have to be extra careful on iOS as you will be more penalized if Apple discovers the bug or the app is launched and you have to quickly fix the bug, ask for expedited review and pray the Apple gods to approve it on the same day. One of the companies mentioned that they asked for 3 expedited reviews on the same week. Apple approved all of them. As the TV ads say, don’t try this at home ;)
Xavier Bisbal (Forcemanager) added that if you have professional clients you have to take into account that they may have old devices. So, you could be developing an app for tablets but after launching they start complaining that it doesn’t work for their iPads 1. They probably aren’t going to buy new iPads; so do you adapt or do you lose a customer?
How to test the apps in situations with low connectivity? Wuaki has a room in their office with poor connection, that’s where they test them. Forcemanager uses the elevator to test their apps. They also went to the airport and they started driving around connecting their phones to the WiFI; and because of its lack of quality it was a perfect real world condition. Marc Canaleta (CTO of SocialPoint) told us a different problem; a user may start playing a level on the iPhone while he is offline and some hours later he moves to the iPad; but they haven’t been able to synchronize the progress. As you can see, there is a lot of imagination here. Luckily there are more and more platforms to test online different phones with different connectivity qualities so developers don’t need to move or buy dozens of phones.
But at the end, it’s true that we throw our apps to users and they are ones who test them.
Mobile — New features
Another topic of discussion was the need of synchronizing the launch of new functionality in the web and the apps. In general, most companies do it only when it’s a new feature. Whereas SocialPoint updates its apps just once per month so they have more time to test everything and plan the launch accordingly.
Xavier Bisbal (Forcemanager, a B2B company) commented that they launched the new functionalities on iOS first because when selling, they talk to bosses and they use iOS. They have stopped doing it because their clients’ employees use Android and they were confused on why some features were in iOS but not yet in Android.
Synchronizing is in fact a challenge because of Apple and its review process that can take over one week. Jordi Miró (Wuaki) added that they have to take into account also the Xbox platform, which only releases its apps on Tuesdays; so there you have another complication.
Connected to this is the possibility of using one mobile platform to test changes or new features. On both of my companies I’ve used the Android app to do A/B tests and try new functionality before implementing it on iOS. Why? It’s faster to develop in Android and you can upload a version every other day (with a limit to not infuriate your users). On one week you can upload 3 versions of the Android app with different A/B tests and get results. On iOS, you will still be waiting for approval.
Mobile — Developing for other platforms
A couple of years ago Windows Phone, Blackberry and Nokia paid developers the cost of developing their apps for their platforms. They wanted to get as many as the top apps as possible to try to avoid the chicken-and-egg problem of users not wanting to change to their devices because there weren’t enough apps and developers not wanting to build them because there weren’t enough users. That strategy didn’t work for Blackberry and Nokia; and we still have to see if it will pay off for Windows Phone. In fact, what has happened is that many companies, who launched their apps on those platforms under those premises, have removed their apps because they have no one to maintain them and they don’t want to pay an external company. There aren’t enough users to justify it.
Mobile — Native vs Hybrid
All companies who shared their experience had native apps, one of them (Cabify) using RubyMotion to share code with the web (models, utilities). Several of them had sections with web-views. They did this in order to release the app sooner or allowing them to A/B test quickly. 8fit has done this since the beginning allowing them to iterate crazy fast. On Selltag the page to add your shipping address and pay was a web view. We did that in order to launch faster on Android and iOS, and to A/B test quickly. I believe it’s a good idea more startups should consider. It has its challenges of course, as the user may notice is not native and it could expose security risks (in our case, we had to implement a token solution to load the URL).
Another company said that they were considering using the back-end to modify the behavior of their apps without needing an update. I already know another company that currently takes this approach, modifying even the look and feel of different parts of the apps. According to them this complies with Apple and Google store policies, but I would be careful.
Mobile — API
There are several approaches to this topic depending on each company. Some have a legacy website with an API, so their mobile apps had to adapt. Others have a private API so they can do non-REST calls in order to do multiple things at once achieving more speed for their apps. And then we have companies with a REST API, public or not.
For example, SocialPoint has a private API on which each version has a six month expiration date. That means that if you download one of their games and don’t update it for six months, it will stop working warning you to update. Idealista has different APIs because they want their apps to keep working with old operative systems. They recently removed their support for iOS6.
I am always more inclined to comply with the REST principles because it creates a framework of order and cleanliness on both API and apps. But it depends on each company and its legacy.
Continuous integration
Florian Motlik (CTO of Codeship) shared their beliefs about how a startup should organize its software development processes.
- Test to make users happy. The better the QA is and the more tests you have, less bugs users will suffer.
- Let your repository drive your development. When you are ready to deploy, developers merge the code into the production branch and then deploy the code with a tool like Capistrano. It can be an stressful moment because they wonder about what it can break. The idea is that the CI system deploys the code to production automatically when the code is merged to the production branch.
- Build an immutable infrastructure. We usually have one or several servers with the code that keep working continuously. The idea they propose is to deploy new servers each time you deploy to production. That way if something breaks you have the old ones untouched, and you have your deploy process ready for scale. It won’t happen that you forget to install a new library on the second library. Once you have checked the new deploy is working fine, you can kill the previous servers.
- Measure & Log everything.
- Turn off access to your machines. Don’t allow anyone (or almost anyone) to access them. That way they can’t install libraries or modify code by hand and everything follows a process. This way you make sure you have the needed tools to measure and log everything. It’s a controversial point of view. I wonder what other people think about it.
Externalize development
Sam Lown (Cabify) and Felipe Talavera (PeerTransfer) think you should externalize development until you have proved your hypothesis and you have the team to continue that job inside. Cabify even externalizes its database, they pay a company that manages it for them. That way they can focus on the value they create to their users, and they can keep growing fast with a small development team. For example, despite growing so fast they have just one part-time iOS developer.
I have doubts about externalizing as I believe you add more points of failure and more communication problems. As they have proved, it can work; but it’s not the common thought between startups. Many investors don’t like if the tech isn’t built inside, because it reduces the value of the startup. I suppose on each stage you can take different decisions. But it’s true that we already externalize many services (email, payments, auth, hosting, maps…).
Coding standards and code reviews
You should follow a coding standard for each language and reject the pull-requests that fail to comply. Developers will go slower on the first weeks, but after that, everyone will advance quicker because they are used to the same style.
Developers love to participate in code reviews, so you should limit the number of reviewers to achieve effectiveness. And of course, write tests. The more developers you have, the more tests you need to have because the code will be more complex. Also, invest in tooling to manage the infrastructure, new developers will be more comfortable.
Fire Drills
John Carvajal (Kantox) explained to us that in order to have a really solid system, from time to time they kill servers in order to check that the system auto-recovers or the sys-admins are able to fix it quickly. Netflix is famous for this approach, they have their own Chaos Monkeys.
TDD
This development methodology is more or less important depending on the sector of the startup. In Kantox they have a zero bug policy, as they deal with transfers of hundreds of million euros. You don’t want to have any problem because it can kill you. In fact, each new employee is told the story of an American bank which once deployed new code to production with a bug, and it had to declared bankruptcy 45 minutes later.
It's worth reading Rodrigo’s short article with his 15 lessons.