freistil News

###Preparing for Dublin

The european Drupal Dev Days will be one of, if not the most important events for us this year. Since they’re only a few weeks away, we’re preparing for the developer meet-up on several fronts. Markus is in charge of the marketing side and he’s been busy this week creating a handout that properly conveys why Drupal developers want to use freistilbox as their hosting platform. Jochen will cover the technological aspect. This week, he started not only preparing a talk about replicable development environments but also building one that can be installed with a single command. We’re going to start a beta test soon; drop us a note if you’re interested!

###Documentation

There isn’t a day when we don’t realise that there’s still too much knowledge buried in our heads instead of being readily available to the rest of the team. We’ve made it an unwritten rule to immediately add a snippet to our company wiki when something like that comes up. (And yes, we should put this rule in writing, too…) This week, we’ve started a new wiki section named “Service Runbooks”. For every service we use (for example, our backup system), we aim to have a complete documentation of its purpose, its actual implementation, standard operating procedures, caveats, troubleshooting hints and references to further information. A great help in building this kind of documentation quickly is the ability of Confluence to define pre-filled template pages.

###Stats

Help Center: This week, we received 25 new tickets and resolved 25, leaving us with 61 open support requests.

That’s it for this week. Have a great weekend, everyone!

• Read more

The Drupal Dev Days in Dublin are coming up and we are very happy to be gold sponsor of this great event. We’ve started working on our own session and are excited to present a great development tool for freistilbox users at the conference!

Last year in Barcelona, I did a talk about “ Staging with git and Drush”. The video recording of this talk got published a couple of days ago together with my slides:

Staging with git and drush from Asociación Drupal on Vimeo.

• Read more

Recently, I attended the Open Source Datacenter Conference, short OSDC, in Nuremberg. This is a short review of my experience.

The three-day conference is hosted by Netways, an IT services company specialised in Open Source Software, just like ourselves.

I arrived on Tuesday night at about 20:30. Dinner was arranged so that even late arrivals could still get something to eat. Even better was that I found five ex-colleagues back from the good old WEB.DE days gathered around a table. A great start to a great conference.

To be exact, the conference had already started in the morning with workshops. But since I didn’t attend them, I can’t say anything about these.

As the conference programme was structured into the tracks “Cloud & Big Data”, “DevOps and Methods” and “Infrastructure Services”, it covered three of the hottest topics in IT at the moment.

On Wednesday, I attended these talks:*“2000 databases later”: A great look behind the scenes at Booking.com, especially their huge MySQL installation. Kris Köhntopp is a brilliant IT operations expert and his opening session, both informative and entertaining, set the bar to a height that IMHO no subsequent speaker was able to reach.

• “Introduction to NoSQL with Couchbase 2.0”: CouchBase promises to be an in-place replacement for Memcached which we’re using heavily for freistilbox. But its lack of redundancy is a problem, so I was looking forward to Tugdual Grall’s talk. I left it confident that CouchBase is worth doing a proof of concept installation.
• “Petabyte storage with Ceph”: As storage technology is another essential topic at freistil IT, attending Martin Gerhard Loschwitz’s talk was a no-brainer. Ceph is getting more interesting every time I hear about it but I don’t think we’ll be using it anytime soon.
• “Configuration management and Linux packages”: The approach of distributing service configurations by auto-generating and deploying config packages certainly is better than doing it manually. But since, compared for example with Chef, it solves only a small part of the system management problem, Schlomo Schapiro could not convince me that this is a way to go in 2013.
• Jan Doberstein finished the first conference day with “The truth is in the logs”. His overview of useful log management tools confirmed that we need to get Logstash running ASAP.

In the evening, we left the hotel together for dinner and drinks at the “Indabahn” bar. I spent most of the time chatting with my ex-colleagues (when they weren’t occupied with playing Ingress) about what’s going on in our jobs and lives.

After some good night’s sleep (note to self: make sure to switch off the AC first thing after getting into hotel rooms in the future), I started refreshed into the second half of the conference:*“lxc@libvirt”: We’re already using LXC for freistilbox and I hoped to perhaps catch a few new tidbits. But not only did Erkan Yanar only scratch the surface, his way of presenting also seemed to me unstructured and improvised, so I left his session disappointed.

• “PostgreSQL in 2013”: While we’re using MySQL almost exclusively, PostgreSQL is gaining more and more traction (for example as the main RDBMS at Heroku) and we’ll need to run it in production for the Chef v11 release. So Michael Renner’s overview over recent developments in the PostgreSQL world provided a good way for me to get up-to-date.
• “Introduction into Hadoop”: Olivier Renault’s session was another opportunity to learn more about a topic I’ve got no personal experience with.

In the evening, I left OSDC 2013 with a lot of new ideas, many of them spawned by simple side remarks of speakers or experiences shared by other participants. That’s what makes OSDC for me worth attending every time, even if I’m already familiar with many of the topics presented there. I’m going to propose a session myself again next year.

Since there were always two talks at a time, I could not attend all the sessions I had been interested in. Sometimes, I also gave preference to the “hallway track”, ie. talking to other conference attendees outside. I’m going to complete my knowledge intake as soon as the session video recordings are available.

The only downside for me was the long train ride to and from Nuremberg. This will change next year when I’ll be able to fly directly to Berlin where the conference will take place from 2014 on. See you there!

PS: There are just too many interesting IT conferences and DevOps meetups for our small team to attend! Help us!

• Read more

Jochen has been appointed DevOps track chair for this year’s DrupalCon Prague. He’s been busy this week with collecting candidates for the track’s featured talk slot.

Because we’re still not nearly as strong on the sales and marketing front as with technical topics, we got us some outside help. On Wednesday, we met with our coach Ute in Frankfurt for a sales and marketing workshop. She helped us tremendously with improving our processes, for example regarding follow-ups on quotes. We’ve got ourselves a long list of homework that will keep us busy over the next few weeks.

With our office assistant Susanne, we’ve streamlined our accounting processes, which at least those customers might have noticed that have overdue invoices.

We’ve also improved our internal communication. Because a daily standup call at a fixed time doesn’t fit with our anywhere/anytime work style, we decided to move it to email. Now, we’re getting an automated email in the morning asking for a retrospective of the previous work day, an outlook on today’s work, obstacles that could cause problems, and suggestions for improving our work. Everyone answers with their personal list of things and is encouraged to start discussions by responding to a coworker’s status. This asynchronous daily exchange has proven far less intrusive than a scheduled conference call. It’s at the same time more efficient because status mails tend to be quite concise while our calls often deteriorated quickly into smalltalk. And not to worry – we still have enough face time in hangouts on focused topics, so we don’t run the risk of becoming estranged to each other.

Have a great weekend, everyone!

• Read more

3 times 52 is 156 – hey, we’ve had our three-year anniversary just recently! Shouldn’t there have been a celebration or something?

From Tuesday to Thursday, Jochen visited Nuremberg for the Open Source Datacenter Conference (OSDC) where he’s been a regular attendant and speaker for some years now. OSDC is a great conference to learn about interesting open source software projects for IT operations and to exchange experiences with other professionals using OSS in the datacenter. Jochen came back with a long list of ideas how we can improve both our daily work and the services we offer to our customers.

For the rest of the week, Jochen had his focus on our recruiting. We’ve received some interesting new applications for our sysadmin job offer and we’ll make sure not to delay checking if they’re a good fit.

Meanwhile, Markus took care of our existing IT infrastructure and our customers. Behind the scenes, he worked on adding geo-replication to our storage clusters which will come in handy for a number of purposes when it’s ready for production. As he’s getting more involved in our marketing, he also made some preparations for the Drupal Dev Days which we’re supporting as Gold Sponsor.

Next week, we’re both going to meet with a business coach for a one-day sales and marketing workshop. Getting external support on running our company is something we still need to get used to and we’re excited to see what results this workshop will bring.

• Read more

On many hosting platforms, including our own DrupalCONCEPT, secure traffic that is encrypted via SSL has to be handled directly by the web server. This not only puts additional computing load on those servers, it also prevents HTTP caching which means less responsiveness. To speed up the delivery of static page assets, some customers choose to use “mixed mode”, i.e. deliver these assets via HTTP even if the page is requested via SSL. But because this workaround can cause sensitive data to be transferred in an insecure way, it is not a practice we recommend.

For freistilbox, we eliminated this shortcoming! If you want to add SSL encryption to a website hosted on freistilbox, we have a great feature for you: SSL offloading. This means that SSL packets are decrypted the moment they reach our freistilbox infrastructure. The content of these SSL packets is then passed on to the next system layers as plain HTTP requests. This has several advantages.

First, content caching works both for plain HTTP and for SSL traffic. Since the Varnish cache proxy is located between the SSL offloading layer and your freistilboxes, it can store static assets and even pages regardless of encryption. You really don’t need to unsettle your visitors with those “mixed content” browser warnings.

The second benefit of SSL offloading is made obvious by its name: Your web application servers don’t have to use precious computing resources for decrypting requests and encrypting responses. Our hosting platform takes complete care of that. (As usual with freistilbox, I can’t resist to add.)

So go on, make your website more secure and enable SSL! You’ll find everything you need to set up SSL in our online documentation.

• Read more

Just a short message while I’m packing my bag for this year’s Open Source Datacenter Conference: If you’re going to be in Nürnberg on Wednesday or Thursday as well, I’d be happy to meet you! Let’s talk DevOps over some coffee or a drink at the bar!

• Read more

Our freistilbox hosting platform is built from the ground up with high availability in mind. In order to minimize the impact of failures, every backend service (i.e. each MySQL database, each Apache Solr core etc.) is running on at least two servers. And if you run your website on more than a single freistilbox, you’re in good shape on the web application level, too.

Redundancy alone doesn’t guarantee maximum uptime, though. Recently, we had to deal with various kinds of network problems ranging from minor packet loss to a full loss of external connectivity. While we can’t prevent datacenter staff from mistakenly shutting down our IP addresses on the routing level, we realized that we needed to make our infrastructure more resilient against other, more common, network issues.

We found that even smaller network congestions, oftentimes caused by high traffic from or to a neighboring server of another datacenter customer, could seriously impact requests from our web boxes to backend services. The reason for this is that, on a box doing hundreds or even thousands of database requests per second, increases of only a few milliseconds in network latency add up quickly. This can very well impact operation to the extent that the box becomes incapable of serving new incoming requests because it runs full with web server processes waiting for their data.

This problem would be even more severe if, instead of leasing “bare-metal servers”, we were using cloud-based infrastructure where we can’t even influence with whom we’re sharing a VM host. The Drupal experts at 2bits even make this recommendation to VPS users:>When you encounter variable performance or poor performance, before wasting time on troubleshooting that may not lead anywhere, it is worthwhile to contact your host, and ask for your VPS to be moved to a different physical server. Doing so most likely will solve the issue, since you effectively have a different set of housemates.

With IaaS vendors like Amazon, that would mean replacing your server instances with others on a trial-and-error basis. What a pain.

To minimize the impact of network performance degradation on our hosting infrastructure, we’ve started three improvement projects:

• Optimize request distribution at the loadbalancer level.
• Build our own CDN.
• Move our servers into dedicated racks.

We did already finish project 1. A loadbalancer needs to distribute HTTP requests to those backend boxes that have the necessary resources and are responsive. Boxes that are maxed out or do not respond for other reasons become ineligible. We recently optimized the health checks that our loadbalancers use to determine what boxes are ready to receive requests. Now, a box only gets passed HTTP requests if it proved itself to be stable by successfully responding to a continuous series of health checks.

One cause of boxes to become unresponsive is that their backend requests “get stuck” on the network. And since we don’t control the network layer, we instead chose to minimize our dependency on it. That’s why, in project 2, we’re building our own Content Delivery Network. We’re going to cover this topic in another blog post, so stay tuned!

Where we still need to rely on the communication with backend services (for example, with database clusters), we need to make this communication more robust. That’s the goal of project 3. We are going to move our servers into our own racks where they share a direct network connection only with each other, not with other datacenter customers. This dedicated network connection makes data transfers between our servers faster, more reliable and more secure.

These are only the most prominent ones of all changes that we’re doing day in, day out to improve the performance and availability of our freistilbox hosting platform. And although the quality of our services is growing steadily, our prices don’t. So, if you know someone who’s looking for a hosting service that reduces their IT headaches without breaking the bank, please tell them about us!

And if you’d like to help us improve our next-generation managed hosting, join the team!

• Read more