freistil News

Until now, our backup strategy didn’t allow for an easy restore of single MySQL databases. Customers that needed to have their database restored sometimes had to endure one or two hours until their website was reset to its previous state again. The reason was that we had chosen the backup method of making a consistent snapshot of the complete MySQL server. So, in order to restore one or more single databases, we first had to restore this snapshot to a spare server where we then were able to dump the single databases that we actually needed.

We’re happy to announce that this weakness has been resolved! Our database backup now works in two phases:

1. Write a database dump for every single database on the server. These dumps are stored locally in generations (daily/weekly/monthly).
2. Copy the dump files to our well-proven enterprise backup system.

With this new strategy, we have stored all database content at several ages and in multiple places and are still able to quickly restore a single database right from its most recent dump file.

We’re optimistic that this change will significantly shorten the time-to-resolve of our customer’s database restore requests.

• Read more

2012 has been a great year for freistil IT. We’ve been growing as a company, as a team and as individual persons. In this last post for this year, I’d like to share a few of the things that we look back to in gratitude.

###Culture

When I started working full-time on freistil IT in 2010, I intended to build a team that solves serious IT headaches for its customers and has lots of fun doing it. I have the feeling that we finally reached this goal this year.

We developed the DrupalCONCEPT managed hosting platform to a mature product that lets our customers run big Drupal websites while having all the necessary IT work done by us. We learned a lot in the process, and in many areas, too:

*DevOps: Keeping our growing IT infrastructure reliable and performant requires continuous work on our architecture. *Entrepreneurship: In order to develop our business strategy, we had to think a lot about what our long-term goals are and with whom we are going to reach them. *Process management: We learned (more often than not the hard way) that to have satisfied customers, a business needs robust processes and efficient tools. *Project management: A good example for the previous point is that, when we realized that we were losing sight of who’s doing what, we decided to introduce the Kanban method together with daily standup meetings.

As a young company in a rapidly changing business sector, we need to learn a lot, quickly. And most importantly, we then need to act on these learnings. Therefore, distributing both work load and knowledge effectively is mission-critical, especially in a decentralized team with different skill sets like ours. This year, we stabilized the foundations for mastering this ongoing challenge.

Our communication infrastructure has now crystallized to a fixed set of tools. While we also use phone and email, most of our communication happens in Campfire and Yammer. We even call the act of going online in the “freistil” chat room “coming to the office”. With Trello, we track our bigger projects (Kanban Style!) while we manage single tasks in Asana. We’re making an effort of documenting as much know-how and “standard operating procedures” as possible in our Confluence wiki. We feel that having a clearly defined place for every important information makes room in our brains for the creative problem-solving our customers need us to do.

Thanks to our strategic work this year, we can honestly say that we love our customers as much as what we do for them. We’re growing with our business and we’re proud of what we achieved this year as a team.

At this point, I have to admit that there were also some growing pains. We had to deal with criticism from customers as well as from ourselves. Our hosting infrastructure suffered some serious outages and performance degradations. In consequence, we implemented what John Allspaw calls a “ blameless post-mortem”: a thorough review process with the single goal of helping the team prevent the problem in the future, instead of just looking back to decide on whom to put the blame. Judging from the fact that the incident rate has been declining continuously over the recent weeks, we’re on a good way here.

###Continuous improvement

These changes in our culture paved the way for important improvements to our infrastructure, products and services.

We gained much better insight into how and what our servers are actually doing by replacing our previous metrics monitoring system with a new one built on Graphite. Every second, we collect hundreds of values from everywhere in our infrastructure. This makes it much easier for us to analyze the incidents that occur and to prevent others while they’re about to happen.

We boosted the performance of our database servers by using Solid State Disks (SSD). These electronic drives may still have far less capacity, but they manage write-intensive databases so much better than their spinning cousins. After extending our backup system, we’re now able to restore single databases within minutes instead of having first to rebuild the whole server on spare hardware.

To use our steadily growing server resources more efficiently, we’ve built our own private cloud infrastructure by combining
Linux Containers with our trusted system management software Chef. Other than with 3rd party cloud products, we keep total control over how resources are distributed by leasing the bare metal from our datacenter partners.

At freistil IT, “infrastructure is code”, which means that everything we build gets automated by writing Chef cookbooks. This year, we’ve made more than 2400 commits to our Chef repository, double of what we did in the first two years combined. We also started to release in-house cookbooks as open source software on our Github account.

###Innovation (and failure)

Looking at the great success of DrupalCONCEPT, we thought intensively about how we could improve our customers’ hosting experience even more. Based on their feedback and our experience running the platform day after day, we created a list of things we could do better. Here are some examples what we found:

*Speed up the deployment of web application updates *Quickly provision additional cluster nodes *Improve the performance and reliability of backend services like databases and storage *Make stage-specific configuration more flexible *Enable customers to use external repositories (e.g. Github) *Fully support external tools like Drush *Open up the platform to other web applications than Drupal

As the result of these deliberations, we decided not to build a new generation of DrupalCONCEPT (which would have been the fourth one in more than 2 years) but a completely new hosting platform. To make it clear that it’s our brainchild, we named it “ freistilbox” and launched the product website in October. Many of our customers were immediately excited by the news and couldn’t wait to get their hands on their own freistilbox cluster. We were literally overwhelmed by this success.

Which brings us to our deepest low of 2012: we failed to deliver on our promise. First, I had dramatically underestimated the effort necessary to build the many small non-incremental improvements from scratch that would distinguish freistilbox from DrupalCONCEPT. Then, we were also forced to divert a big part of our time to solve some serious problems with existing installations. In consequence, our launch schedule fell apart completely. While we were able to bridge the gap by providing DrupalCONCEPT servers, we are devastated knowing that we severely disappointed our most loyal customers. We’re going to make a huge effort in the coming year to compensate for that.

###Perspective

In 2013, the ongoing development of freistilbox will be our main focus. What we’re seeing so far makes us very excited but we’ve only just begun.

Because we’ve clearly reached the limits of what we can achieve with such a small team, we’re going to get additional people on board as quickly as possible. (So, if this whole high-end hosting stuff sounds interesting to you, please get in touch!)

And finally, there’s also going to be growth on the private side: both Markus and I expect new arrivals to our families in early 2013. Looks like we will need all the flexibility that our “Results-Only Work Environment” allows…

##Thank you.

When I look back at 2012, what I feel the most is gratitude. I’m thankful to our awesome customers for encouraging us to keep improving, especially when we fail. I’m thankful for being able to work as part of a great team that values results over time spent “at work”, and creativity over control. Most of all, I’m thankful to my family for giving me their trust and love.

And finally, thank you for reading this long. ;-)

To a brilliant 2013!

• Read more

From this Saturday (2012-12-22) on, our team will be off recharging.

During the holiday time, we’ll only do emergency support . That means we’ll only handle outages and other incidents that impact the delivery of existing websites.

We’ll resume working on tasks that aren’t connected to such incidents on Monday, January 7th 2013.

So, if you plan to launch a new website over the holidays or need other kinds of assistance, please let us know immediately!

• Read more

In private conversations I’ve already been talking about our “new architecture” for some time, and today, we present it to the world. On behalf of everyone here at freistil IT, I’m very happy to announce the release of freistilbox — our next-generation managed hosting platform!

We’ve taken everything we’ve learned over the past three years to build this platform. We’ll have a technical article for those interested soon. Until then, here are the big benefits of our freistilbox hosting platform!

Scalability made simple

freistilbox is built to scale without forcing our customers to deal with the complexity that comes with a growing hosting infrastructure. To get more capacity (and higher availability as a side benefit), you just add more “boxes” to your hosting platform. These boxes are dedicated server instances that host your web application. Everything else — databases, caches etc. — is managed by our operations team behind the scenes.

Load Distribution

Only our DrupalCONCEPT ELITE customers had access to a load balancer that distributes incoming requests between two or more application servers.

For freistilbox, we’ve built load distribution right into the core infrastructure, so as soon as you have two or more freistilbox instances, incoming requests automatically get distributed between them. Should one of these freistilbox instances suffer an outage, it will automatically be left out, so your website visitors won’t notice.

SSL offloading

With DrupalCONCEPT, while answers to HTTP requests are answered very fast by a Varnish cache, SSL-encrypted traffic gets delivered directly to the web application servers. There, the requests have to be first decrypted and then answered by Drupal — every single time, because we are going around the Varnish cache. That’s why SSL-heavy sites tend to be slower and need a lot of processing power.

On our new freistilbox infrastructure, SSL requests are decrypted immediately when they arrive by a dedicated SSL offloading layer. This not only takes computing load off the application servers but also allows the HTTP cache to speed up both encrypted and unencrypted traffic.

Integration of external repositories

In DrupalCONCEPT, the Git repository containing the web application code is tightly integrated with the website instance itself. We use Git for freistilbox, too. But if you already have a Git repository that supports webhooks, for example at Github, you can use that one to deploy your code instead of the default repo we provide.

Deployments take less than 10 seconds

With DrupalCONCEPT, if you wanted a site update rolled out, it would take up to 2 minutes for the deployment mechanism to do its thing. Now, thanks to a newly designed deployment process, it takes less than 10 seconds.

Web dashboard and API

Not only does freistilbox offer a web-based dashboard where you can see (and, in later versions, change) your customer data and website configuration. The dashboard application provides a RESTful API that we’re also going to document and expose to the public. That means that you will be able to make changes to your platform parameters either manually in your browser or automated via software.

Upcoming Features

There’s no set schedule for these yet, but we know you want them, and we’re testing them. We’re working hard to bring you:

• Separate SSH servers to execute commands like mysqldump or drush manually or via Cron, without using precious application server resources
• Secure external access to your MySQL database Of course, as we know ourselves and our customers, we’ll add many other things over time that we don’t even think about today.

What about DrupalCONCEPT?

No change on this front. We’ll continue operating all DrupalCONCEPT installations that already exist. We’ll just stop adding new ones from today on.

So, what do you think?

We’d love to know what you think of our new hosting platform, so head on over to www.freistilbox.com and make sure to drop us a note in the comments below!

• Read more

Hi there and welcome to freistilbox, our next-generation managed webhosting platform! Everyone here at freistil IT is very excited that we’ve finally reached its launch day!

You can read more about what freistilbox is all about on our company blog. And, of course, just check out the navigation menu above to find all the details!

• Read more

From August 20th to 24th, Drupal users and experts from all over the world visited Munich for DrupalCon Europe 2012. Among them, Markus and I. Since we do what we can to be a good community member, we backed the event as a Bronze Sponsor. And boy, did we get something for our money!

Let me put it first that the organization team did an amazing job, in front of and during the conference. They chose the Westin Grand Hotel and the nearby Sheraton as venues for all the talks, show floors and group meetings. It didn’t take long after arriving to notice that this was a very good choice. Not only was there enough space for almost 2000 conference visitors but the hotel staff also mastered the logistics of providing everyone at any time with coffee, water and food. There seemed to be a food table and coffee stand at every corner and I never saw a queue worth mentioning. The food was so great in terms of both quality and variety that I didn’t dare to get on my scale for the following week. There were a few weak points, too, for example the badly ventilated “Garmisch” conference room in the basement.

Another great idea was to merge the conference bagde with the program guide. If you wanted to see what was going to be on next, you conveniently flipped open the booklet on your chest.

The most important part of DrupalCon is, of course, the sessions. This time, they were divided into the following topic tracks:*Coding and Development

• Community
• Frontend
• Business and Strategy
• Sitebuilding
• DevOps

Additionally, you could take part in the Core Conversations discussing Drupal’s future and watch live discussions at the Day Stage. In sum, visitors could choose between 78 sessions. All of them (AFAIK) were recorded and the A/V team managed to put the recordings online within impressively short time.

I’m proud to have given two sessions myself: “ Simple devops workflows with Kanban” and “ Use datacenter tools to make your dev life easier”. I’m going to write some blog posts following up these sessions soon.

Speaker support by the DrupalCon team had been impressivly intense. Not only did they make sure that the speakers finish their materials in time but they also offered several webinars to help the speakers ramp up their presentation skills. The quality of the sessions I visited was quite mixed, though, and many people I spoke to wished for more pro-level topics. I think it’s a reasonable demand that DrupalCon sessions meet a higher standard than those at local DrupalCamps.

On the vendor show floors, most companies had chosen the dull and boring approach of having a more or less sophisticated booth where people handed out stuff and talked to visitors. Two of the few notable exceptions were propeople with their giant Jenga tower of Drupal components and Comm-Press who handed out T-shirts with QR codes that matched in pairs. Those who made the effort to find the Drupalista wearing the matching shirt could win an iPad or tickets for next year’s DrupalCon Portland.

There was a lot of buzz about the merger announcement of four well-known Drupal shops. Like everyone else, I was amazed that this made them the world’s biggest Drupal development business and surprised that they chose to keep the name of the smallest business, the german “Wunderkraut” (which translates, unbeknownst to many, as “wonder herb”). Looking back on my experience with big mergers, I wish them best of luck with the challenges they’re facing, especially the essential task of building a common company culture.

I’d like to end with shout outs to some people I enjoyed spending time with: Gerhard and Kris – thanks for having me on the DevOps track and for all the support! Micha, Thomas, Jan – thanks for a great evening with Pizza and late desserts! Barry and Josh – I had a lot of fun with my fellow Drupal hosting experts at the DevOps Meetup Munich and in the English Garden!

And finally, a big Thank You to the Drupal Association for organizing such a great event! I’m already looking forward to the next DrupalCon!

• Read more

It’s been 3 weeks now, but I still wanted to write and say thanks for making Drupal Camping the way it was!

It were 5 days of Drupal, great people, sunny weather, hard rain, interesting sessions and a generally a very nice place to be at.

Ralf and Pascal wrote about the event. You can find some photos on the Facebook Page and in Tobias Flickr set.

And just 3 days ago the german Drupal podcast DCPod published DrupalCampingKompottCast, featuring some Drupalcampers and Kars-T and Caseledde playing the official song.

Thank you for being so great, Drupal community :-D

And thanks a lot to comm-press for making this happen.

See you next year!

• Read more

A few weeks ago, we put our first MySQL cluster equipped with solid state disks (SSD) into production. It turned out to be a great improvement to our hosting platform, so from now on, we’ll put all new databases of our DrupalCONCEPT ELITE clusters on SSD-based database clusters!

An SSD is basically electronic memory with a disk interface so you can use it like a normal disk drive. But since it doesn’t need to move a head over a spinning magnetic surface to access its contents, it is faster than a normal harddisk by orders of magnitude.

This is a great advantage, especially for write access. On our harddisk-based MySQL clusters, we can boost read access by caching data in server memory at the first request and serving it from there for subsequent requests. Write requests, though, need to always go directly to the storage device to prevent data loss in the case of a sudden server crash. Normal hard disks have decent performance writing big chunks of data but look quite bad when they have to swing their heads all over the place for random write accesses. Unfortunately, most write requests on database machines actually are small changes at random places. The common way of mitigating this performance degradation is by combining many harddisks to a RAID array. The goal in this case is not to combine their storage capacity but to get as many independently moving disks heads as possible. Since, in this case, a RAID–10 configuration (data protection by mirroring complete disks) is superior to RAID–5 (data protection by calculating checksums), building a high-performance database server with harddisks becomes quite expensive.

By using SSDs as database storage, we can solve the performance problem quite elegantly. We combine a few of them into a RAID–10 array and add two normal harddisks in a RAID–1 configuration for data that is written in contiguous chunks, for example log files. The result is astonishing database performance (both read and write), paired with the still necessary protection from data loss.

And this awesome performance and data protection will be available to all our DrupalCONCEPT ELITE customers – both new and existing – from now on. And at no extra cost, too!

Too good to be true? Well, there’s actually one downside: If your application does some kind of insane database query that produces gigabytes of data, our new database clusters will deliver them so quickly that it will probably eat up your server’s network bandwidth. So tame those monster queries, folks! ;-)

If we already provide you with a DrupalCONCEPT ELITE cluster and you would like to migrate your existing databases to a SSD-based database cluster, please drop us a line. If you are interested in what makes DrupalCONCEPT ELITE one of the best Drupal hosting platforms in the world, check out the DrupalCONCEPT website!

• Read more