2012 in review
Published 2012-12-31 by Jochen Lillich
2012 has been a great year for freistil IT. We’ve been growing as a company, as a team and as individual persons. In this last post for this year, I’d like to share a few of the things that we look back to in gratitude.
###Culture
When I started working full-time on freistil IT in 2010, I intended to build a team that solves serious IT headaches for its customers and has lots of fun doing it. I have the feeling that we finally reached this goal this year.
We developed the DrupalCONCEPT managed hosting platform to a mature product that lets our customers run big Drupal websites while having all the necessary IT work done by us. We learned a lot in the process, and in many areas, too:
*DevOps: Keeping our growing IT infrastructure reliable and performant requires continuous work on our architecture. *Entrepreneurship: In order to develop our business strategy, we had to think a lot about what our long-term goals are and with whom we are going to reach them. *Process management: We learned (more often than not the hard way) that to have satisfied customers, a business needs robust processes and efficient tools. *Project management: A good example for the previous point is that, when we realized that we were losing sight of who’s doing what, we decided to introduce the Kanban method together with daily standup meetings.
As a young company in a rapidly changing business sector, we need to learn a lot, quickly. And most importantly, we then need to act on these learnings. Therefore, distributing both work load and knowledge effectively is mission-critical, especially in a decentralized team with different skill sets like ours. This year, we stabilized the foundations for mastering this ongoing challenge.
Our communication infrastructure has now crystallized to a fixed set of tools. While we also use phone and email, most of our communication happens in Campfire and Yammer. We even call the act of going online in the “freistil” chat room “coming to the office”. With Trello, we track our bigger projects (Kanban Style!) while we manage single tasks in Asana. We’re making an effort of documenting as much know-how and “standard operating procedures” as possible in our Confluence wiki. We feel that having a clearly defined place for every important information makes room in our brains for the creative problem-solving our customers need us to do.
Thanks to our strategic work this year, we can honestly say that we love our customers as much as what we do for them. We’re growing with our business and we’re proud of what we achieved this year as a team.
At this point, I have to admit that there were also some growing pains. We had to deal with criticism from customers as well as from ourselves. Our hosting infrastructure suffered some serious outages and performance degradations. In consequence, we implemented what John Allspaw calls a “ blameless post-mortem”: a thorough review process with the single goal of helping the team prevent the problem in the future, instead of just looking back to decide on whom to put the blame. Judging from the fact that the incident rate has been declining continuously over the recent weeks, we’re on a good way here.
###Continuous improvement
These changes in our culture paved the way for important improvements to our infrastructure, products and services.
We gained much better insight into how and what our servers are actually doing by replacing our previous metrics monitoring system with a new one built on Graphite. Every second, we collect hundreds of values from everywhere in our infrastructure. This makes it much easier for us to analyze the incidents that occur and to prevent others while they’re about to happen.
We boosted the performance of our database servers by using Solid State Disks (SSD). These electronic drives may still have far less capacity, but they manage write-intensive databases so much better than their spinning cousins. After extending our backup system, we’re now able to restore single databases within minutes instead of having first to rebuild the whole server on spare hardware.
To use our steadily growing server resources more efficiently, we’ve built our own private cloud infrastructure by combining
Linux Containers with our trusted system management software
Chef. Other than with 3rd party cloud products, we keep total control over how resources are distributed by leasing the bare metal from our datacenter partners.
At freistil IT, “infrastructure is code”, which means that everything we build gets automated by writing Chef cookbooks. This year, we’ve made more than 2400 commits to our Chef repository, double of what we did in the first two years combined. We also started to release in-house cookbooks as open source software on our Github account.
###Innovation (and failure)
Looking at the great success of DrupalCONCEPT, we thought intensively about how we could improve our customers’ hosting experience even more. Based on their feedback and our experience running the platform day after day, we created a list of things we could do better. Here are some examples what we found:
*Speed up the deployment of web application updates *Quickly provision additional cluster nodes *Improve the performance and reliability of backend services like databases and storage *Make stage-specific configuration more flexible *Enable customers to use external repositories (e.g. Github) *Fully support external tools like Drush *Open up the platform to other web applications than Drupal
As the result of these deliberations, we decided not to build a new generation of DrupalCONCEPT (which would have been the fourth one in more than 2 years) but a completely new hosting platform. To make it clear that it’s our brainchild, we named it “ freistilbox” and launched the product website in October. Many of our customers were immediately excited by the news and couldn’t wait to get their hands on their own freistilbox cluster. We were literally overwhelmed by this success.
Which brings us to our deepest low of 2012: we failed to deliver on our promise. First, I had dramatically underestimated the effort necessary to build the many small non-incremental improvements from scratch that would distinguish freistilbox from DrupalCONCEPT. Then, we were also forced to divert a big part of our time to solve some serious problems with existing installations. In consequence, our launch schedule fell apart completely. While we were able to bridge the gap by providing DrupalCONCEPT servers, we are devastated knowing that we severely disappointed our most loyal customers. We’re going to make a huge effort in the coming year to compensate for that.
###Perspective
In 2013, the ongoing development of freistilbox will be our main focus. What we’re seeing so far makes us very excited but we’ve only just begun.
Because we’ve clearly reached the limits of what we can achieve with such a small team, we’re going to get additional people on board as quickly as possible. (So, if this whole high-end hosting stuff sounds interesting to you, please get in touch!)
And finally, there’s also going to be growth on the private side: both Markus and I expect new arrivals to our families in early 2013. Looks like we will need all the flexibility that our “Results-Only Work Environment” allows…
##Thank you.
When I look back at 2012, what I feel the most is gratitude. I’m thankful to our awesome customers for encouraging us to keep improving, especially when we fail. I’m thankful for being able to work as part of a great team that values results over time spent “at work”, and creativity over control. Most of all, I’m thankful to my family for giving me their trust and love.
And finally, thank you for reading this long. ;-)
To a brilliant 2013!