devopsy.kinsta.cloud

Author: David Safaii

The Cost of Rework when Developing for OpenStack Clouds
According to the 2016 State of DevOps Report by Puppet and DORA, the mean time to recover when a service incident occurs ranges from less than one hour for high IT performers to less than one day for medium and low IT performers. Put that in a real-world scenario: your OpenStack application and the data associated with it that spans multiple virtual machines failed (for one reason or another). Now you need to rebuild something you’ve already spent time developing. Are you—and your boss—OK with that rework taking hours to days? Or do you think you’d be wishing you fell in the range of high IT performers, and you could quickly recreate what you lost in less than one hour?

Now, take it a step further. What could you be doing with that time you now need to dedicate to rebuilding something? What will that rework take you away from? The next phase of your development? Lunch? Happy hour? Sleep? What price would you put on the personal cost to you?

The 2016 State of DevOps Report authors wanted to calculate the cost of excess rework per year for organizations. They created a formula to calculate the number:

Cost of excess rework = Technical staff size x average salary x benefits multiplier x percentage of technical staff time spent on excess rework

Here’s what they found:
- High performers have 11 percent excess rework
- Medium performers have 22 percent excess rework
- Low performers have 17 percent excess rework
At first glance, those numbers seem odd. Why would the low performers have less excess rework than the medium performers? The report clarified that by explaining “low performers spend less time on rework … because they report spending a greater proportion of time on new work. We believe this greater amount of new work could be occurring at the expense of rework, thus racking up technical debt.”

It’s clear all organizations have some amount of rework as a result of a failure of some kind. The question is, What is the cost of that rework? Let’s take a look at the results of the study.

For the most part, the simple answer is that rework will likely cost your organization millions (unless it’s costing you billions). It’s just a matter of how many millions.

We often get asked why it’s necessary for OpenStack environments to have a data protection strategy and a backup solution. The conventional (but wrong) thinking is that virtualization can take the place of data recovery. Frankly, the answer to why is contained within those numbers above. If you don’t have a backup and recovery solution for your OpenStack development efforts, are you willing to spend money and resources on reworking the lost data? And if you are, how many millions are you willing to spend on it? Let me remind you that you are trying to recover to a point in -time … persistent data included. If you say, “If you built your cloud appropriately, this shouldn’t be an issue ….” Madame or Sir, if your virtualized cloud has that kind of resiliency, then I would like to work for you.
August 18, 2016
The Maturing OpenStack Cloud: Just Do it
We’ve been thinking about the maturing cloud a lot lately. In fact, we’re participating in a panel discussion on the topic at the upcoming OpenStack East in August in New York. As we’ve spoken with OpenStack users and vendors throughout the world we’re hearing a common theme about the maturation of OpenStack clouds—the results seldom look like what was originally planned. Disruptive technologies, changes in personnel, the evolution of business strategies … these are all unplanned events that can change what the mature cloud eventually will look like. The challenge for today’s DevOps teams is to how to maximize the opportunities for success, despite the likelihood of unanticipated changes. After all, as many DevOps folks know, between the design of an application or system and the completion of that project, they often are faced with a change in the initial business requirements of the project.

Lao Tzu famously said, “The journey of a thousand miles begins with one step.” While that is true, we’d add that the journey will be much smoother if that one beginning step relies on careful consideration of the best shoes for the journey: shoes that are comfortable, durable and multi-purpose. Shoes that see you through any type of weather. The right shoes won’t necessarily get you to the right destination, but they certainly will make your journey easier and better ensure you complete your journey—even if you end up somewhere other than where you thought you would.

So what do shoes have to do with the OpenStack cloud? We liken choosing the right shoes to making the right architectural decisions at the beginning of your OpenStack journey. One of the most important architectural decisions DevOps can make early in the development of the OpenStack cloud is to include a comprehensive data protection strategy, accompanied by the appropriate tools to facilitate that. There are three primary reasons to include data protection from the start:
- It’s a “given” for VMWare and other enterprise applications for a reason. One might argue there’s no need for data protection, since everything lives in the cloud. But we all know files, applications and systems can fail for no apparent reason. It’s much easier to recover from a failure if you have a data protection strategy in place, rather than having to suddenly (and often quickly) re-create the lost resources.
- It will make it easier to manage OpenStack and cloud resources. As your OpenStack cloud grows, you may need to move workloads to different availability zones or data centers. While you may have a general idea of your resources needs at the beginning of the DevOps process, chances are things will change along the way.
- It makes for a smoother upgrade process. Chances are that you are not rolling out your OpenStack cloud with the exact OpenStack version you started with. For example, one of our customers started on Kilo and now is migrating to Mitaka for its rollout. Their migration is much easier because the customer knows everything is backed up, available to use and restored in its new, upgraded environment.
Sure, the OpenStack and the cloud paradigm is enabling DevOps and business users to upend many “traditional” approaches to IT. But that doesn’t mean you should toss out every strategy that’s considered “traditional.” After all, if you were buying your shoes for your thousand-mile journey, would you only buy one pair? Or make sure you have a spare just in case?
August 1, 2016
Why Legacy Backup Solutions Aren’t Ideal for OpenStack
Our team has recently returned from the OpenStack Summit in Austin. While there were many hot topics, we discovered people fell within two camps when it comes to backup and recovery: those who understand why a native OpenStack solution is required, and those who question why legacy backup and recovery won’t fit the bill.

I sat down with Trilio Data’s co-founder, Murali Balcha, to detail why legacy backup and recovery solutions are not suited to protect OpenStack applications without considerable retooling. Even then, these solutions do not naturally fit into OpenStack operating model and will increase your total cost of ownership (TCO) considerably.

First, let’s look at the OpenStack operating model. It includes:

Horizontal Scale

OpenStack does not come with the maximum capacity that it can grow into. The architecture is inherently scalable without any theoretical boundaries; it can be deployed in a single box or it can scale to thousands of nodes.

Contrast this with existing backup solutions that always have some predefined capacity, whether it is number of applications it can back up or the amount of backup storage capacity that it can manage. So essentially you have a forever scalable architecture, tied to a backup solution with a limit on capacity.

Multi-Tenancy

Multi-tenancy is an integral part of OpenStack cloud. The tenancy is different than supporting multiple active directory users in a backup application. Active directory users are typically OS users, whereas cloud tenancy usually represents individual organizational units with in a bigger organization. So there is a mismatch on how existing backup solutions are created versus how OpenStack multi-tenancy works.

One way to make existing solutions work with OpenStack is to deploy a backup application for each tenant and make that tenant responsible for managing its own instance of the backup application. This means each tenant has to manage the application and any storage that is provisioned to save backup images.

Self-Service Portal

Some backup applications provide an aspect of a self-service portal for individual users to manage their own backup policies. Backup applications use host agents on either virtual machines, or on servers to back up files and provide a portal to restore a backup copy in place through its agent infrastructure. This kind of self-service portal is, again, in contrast to what an OpenStack self-service portal is—it provides the ability to provision virtual resources.

File-Level Backups

Existing backup solutions are only good at doing file-level backups. Yes, they have evolved to do virtual machine (VM) backups for VMware, but vCenter is not a cloud platform, and the mechanism to do a VM backup on VMware platforms is totally different from doing VM backups in OpenStack. Anyone trying to retrofit existing backup solutions for OpenStack will only have their file-level backup feature to rely on.

File-level backups only back up bits and pieces of your application. File-level backups are the legacy of backing up scale-up systems, where huge databases are confined to a beefed-up single host. But cloud applications are lot more distributed. Instead of provisioning one beefed-up server to host a large database, cloud applications are deployed as multiple VMs, with each VM performing discrete functionality. The data in these applications are also are distributed to multiple VMs. File-level backups will not ensure consistent backups. They are notoriously insufficient to restore an entire application environment and may seriously affect your recovery time.

Infrastructure-Aware Backup Solutions

File-level backups are not well-suited to address cloud application needs. Instead, a backup application that is aware of cloud infrastructure provides a perfect solution for your cloud applications. Since cloud applications are distributed, your backup should include not only the data but also the application context to successfully recover the application.

An application context includes the VMs, flavors of the VMs, network settings and all the volumes attached to those VMs. Let’s not forget that all of the aforementioned will be tweaked and will evolve over time to suit that moment in time. So only backup solutions that can provide a single snapshot that includes all the VMs, storage volumes mapped to each VM, network settings and the meta data associated with each VM and its resources at that point in time are suited for OpenStack backup needs. Since such backup solutions capture the entire application environment, recovery with this backup solution is very deterministic and reliable, irrespective of the size and complexity of application.

Solutions that are aware of the infrastructure also can back up applications into one availability zone/region and restore into a different zone/region. Beyond speed of recovery, this now empowers data protection (or backup) with resource management capabilities.

DevOps

DevOps is an integral part of OpenStack cloud deployment and management. Your choice of backup solution should play with your cloud DevOps scripts. For example, if you standardize on Ansible playbooks to manage your cloud deployments, the backup application should play into your DevOps strategy so you can reduce the TCO of your backup application.

Other Use Cases

In traditional IT, the data management functions such as backup and recovery, disaster recovery and application migration require different applications. However, your backup solution for OpenStack should be more than just a backup and restore of the files. A backup solution that is well-designed for OpenStack is a re-orchestration from a point-in-time and should be able to provide additional uses cases such as:
1. File(s) restore
2. VM/volume restore
3. Intra/inter-cloud application migration
4. Disaster recovery
5. Instant restore of point in time for application validity checks, network scanner and other forensics
6. Archival
7. Environmental restore during cloud upgrades
OpenStack offers an operating model that warrants a completely new thinking around backup solutions. Retrofitting existing solutions not only is inadequate in protecting your applications, it also increases your TCO. So do yourself a favor and look for solutions that are purpose-built for OpenStack. That will provide the best data protection possible, decrease your TCO and simplify recovery.
May 30, 2016
Top 5 Reasons to Back Up Your OpenStack Environment
I am a big fan of lists. If you recall the movie “High Fidelity,” John Cusack’s character Rob and his co-workers—armed with an encyclopedic knowledge of all things musical—compile “top five” lists for every conceivable occasion, openly mock the tastes of people, and, every so often, sell a few records. In the spirit of a lists, here are the top five reasons to back up your OpenStack environment. We thought that it would be helpful to many.

If you’re currently working with OpenStack clouds, or are in the process of embracing it, you need to include backup and recovery for OpenStack on your list of to-dos from the start. You may ask, “Why? I have orchestrated/automated everything, I just need to spin resources up and down.” Cloud environments are fundamentally different from the virtualization we have become used to. Virtualization relies on infrastructure for resiliency. In the cloud, distributed applications are responsible for their own resiliency, and are independent of the underlying infrastructure. Therefore, cloud apps need backup and recovery options, lest you face a complete machine, data or file meltdown.

We’ve compiled the top five reasons to backup your OpenStack environment. Take a read and it will become clear why you must include backup in your OpenStack plans.
1. Compliance — Compliance may seem like an issue for only certain industries—finance and health care come to mind—but nearly every organization has to deal with compliance issues. For some, it’s compliance to external regulations, while for others it’s simply complying with internal policies and best practices. Backup for OpenStack is an essential component for compliance; it ensures the availability of the data at a specific point in time within your OpenStack applications. I was recently involved in a dialogue/string on LinkedIn watching experienced OpenStackers go back and forth debating “pet vs. cattle.” “I am getting tired of it,” one person said. “This is cloud. It’s about scaling up and out … With automation and orchestration there is no need for cattle.” This conversation went on for days. Theory is great. Practice is reality. What people failed to discuss is that compliance or the business units will drive many requirements (such as backup) either today or tomorrow. End of story.
2. DevOps — Backup is often the forgotten tool for DevOps. Containers are great to use, but the ability to test with real data is an issue. The reason to use backup is quite simple: test and development. With a copy of a production environment, you now can test upgrades and new features in your agile development process. Furthermore, if you lose a machine or your data, or you end up with corrupted files while in the middle of testing or development, your life will be much easier if you have an appropriate backup solution in place. Restoring machines, data and files is a better option than re-creating them entirely. Cookbooks and recipes are great as well, but you still need to couple it with the appropriate “ingredients.” It is like playing with a fake kitchen as a child … do you want to make breakfast with plastic pancakes and toast? Nothing beats the real thing. Yes, more resources are required for one over the other, but when something goes wrong—it is unfortunate—people point fingers.
3. Disaster recovery plans — While the terms “backup” and “disaster recovery” often are used interchangeably, they are actually not the same thing. You might have what appears to be a great recovery environment (ideally one that mirrors your production environment for high availability), but without a comprehensive backup solution, your recovery might not be the painless experience you expect it to be. Perhaps the solution for intelligent disaster recovery is to marry backup with a Swift environment so you can recover from any point in time, anywhere in your organization.
4. Save time — Often recovery can be a frantic process, driven by a business need to have the machine, data or files recovered as soon as possible. Or, more likely, sooner than possible. If you are trying to re-create something under that type of environment, there’s a likelihood that mistakes could be made. This becomes increasingly true the more complex and mature your cloud becomes (think about the growing number of workloads and applications). A much easier approach—one that provides you with peace of mind—is to have a comprehensive backup solution ready and available for when you need it. There’s also another reason backup can save you time: When you have to re-create the lost information, you are taking your time away from something else—and that means you are taking a big hit to your productivity. Save yourself and your organization a big headache with backup.
5. Security and forensics — You never know when you are going to be asked to provide data or files related to a security issue or a forensics investigation. The right backup solution not only makes it easier to assist with the activity, but think how good you’ll look by being able to deliver the information the forensics team needs quickly.
Are you currently looking into backup solutions for OpenStack? Now that you know why you need backup for OpenStack, learn more about how to backup OpenStack.
March 10, 2016