devopsy.kinsta.cloud

Tag: backup

Enlisting the C-Suite in Disaster Recovery Initiatives
When disasters make headlines, corporate executives often have knee-jerk reactions and call for business continuity solutions to be implemented immediately. However, when push comes to shove, those very same executives tend to put up roadblocks with concerns over budget, priorities, vulnerability assessments and security.

Perhaps more concerning, according to a June 2015 Ponemon Institute report, there are “a large portion of companies who remain vulnerable as a result of lack of executive interest in preparing for worst-case scenarios.” This creates a precarious situation for DevOps professionals that understand the severity and breadth of threats that can trigger downtime—from human error to power outages, cyberattacks and natural disasters.

To expedite disaster recovery initiatives, IT teams must enlist top executives that influence budgeting directly. This often calls for varied approaches that speak to specific executive concerns and issues.

Understand Your Options: Disaster Recovery vs. Backup

It’s very common for companies to confuse disaster recovery and backup solutions. Both are available via cloud, giving teams access to capabilities that were once only available to enterprises with huge IT footprints. When you approach any executive, it’s critical to understand various services and how they can work in tandem to deliver the most cost-effective protection.

Backup ensures companies do not completely lose data because it is backed up with copies. While it is important to have copies of data, accessing and restoring backups can take some time. Therefore, for workloads with important but not extremely time-sensitive data, backup can be sufficient while satisfying budget restrictions. In the fast-paced DevOps world, backup usually falls short.

On the other hand, cloud-based disaster recovery ensures minimal—if any—downtime. Applications can fail over and be up in running in seconds. Further, testing can be done at will without any impact on production environments. For workloads that demand continuity of service and aggressive recovery point objectives (RPOs), disaster recovery solutions are the better option and can also deliver cost efficiencies.

For many operational and regulatory compliance reasons, companies may require a mix of both backup and disaster recovery. When evaluating service providers, check into how easy it is to apply backup and disaster recovery based on your development and application requirements. Take a close look at pricing models associated with each—including any support fees.

CEO/COO – Chief Executive Officer and/or Chief Operations Officer

While CEOs and COOs generally delegate detailed budget decisions to their teams, they are still ultimately responsible for overall company performance and reputation. Anything that threatens ongoing conduct of the business is their purview. Be prepared to discuss risks in real terms, and clarify that a “disaster” can be triggered by something as simple as human error. They will care most about considerations such as:
- Time to business resumption [e.g. near-zero recovery time objective (RTO)]
- Capital investment via pay-as-you-go options
- Ability to prioritize data and workloads, both for budgetary and business continuity purposes
- Data loss estimates, in the case of a disaster
- Testing flexibility, to ensure an operational, reliable solution
- Independent validation from similar customers or analysts that a solution is reliable, proven and cost-effective
CIO – Chief Information Officer

Theoretically, this should be the easiest executive to convince because the CIO has a personal stake in recovery, but remember CIOs have a tough balancing act between controlling budgets and limiting exposure. The 6 Disaster Recovery Doppelgangers quiz and e-book can help CIOs approach their disaster recovery challenges across the organization. CIOs generally focus on:
- Ability to see and manage the recovery process
- Scalability of resources with on-demand access, often honing in on flexible terms
- Security and compliance capabilities and support
- Testing at will to ensure solution performs despite configuration changes to the day-to-day operational environment
CFO – Chief Financial Officer

Everything boils down to financials for CFOs, so they should be approached from a fiscal angle that demonstrates value backed by metrics. Be prepared with unbiased, independent information that appeals to their risk averse nature. What other financial benefits might come from leveraging cloud-based disaster recovery? For example, in some cases DevOps teams are beginning to use the failover cloud as a load testing environment that would otherwise require significant on-premise investments. Disaster recovery-as-a-service (DRaaS) environments with advanced security can also be used to check for possible vulnerabilities to cyberthreats.

CFOs often care about:
- Flexible pricing options that can be tailored at a granular level based on services required
- Pay-as-you-go pricing models that avoid upfront capital investment
- Geographic constraints on their data, particularly if backups or failovers need to be maintained in a specific location due to jurisdiction requirements
- Extended DR use cases that could offset other costs like security testing and quality assurance
CMO – Chief Marketing Officer

On top of branding, advertising and public relations responsibilities, CMOs have key objectives that pertain to customer loyalty and overall corporate reputation. They may not be interested in the details of services, but can be a powerful advocate that can speak to the need for business continuity solutions that will protect brand image, reinforce messages about your technology innovation and maintain excellent customer support. Familiarize yourself with real-world stories that demonstrate how media—both traditional and social—cover outages and their negative impact to business. One need not look further than the recent Delta and Southwest outages.

The concept of DevOps is about culture as much as it is about technology, and communication and collaboration are key. By understanding the needs and responsibilities of all stakeholders involved, teams can cut through politics, efficiently protect business and begin to leverage cloud-based DR technologies for initiatives far beyond recovery, alone.

About the Author / Lilac Schoenbeck

Lilac Schoenbeck is vice president of Product Management and Marketing at iLand. She has nearly 20 years of experience with product management, marketing, strategy, business development and software engineering in the grid, virtualization and cloud domains. She is currently vice president of marketing at iland, a global enterprise cloud hosting provider. Prior to this role, she led cloud and automation marketing for BMC Software and has worked for IBM, Fortisphere, Innosight and the Globus Alliance. Lilac holds an MBA from MIT Sloan School of Management and a Computer Science degree from Pacific Lutheran University.
September 14, 2016
The Cost of Rework when Developing for OpenStack Clouds
According to the 2016 State of DevOps Report by Puppet and DORA, the mean time to recover when a service incident occurs ranges from less than one hour for high IT performers to less than one day for medium and low IT performers. Put that in a real-world scenario: your OpenStack application and the data associated with it that spans multiple virtual machines failed (for one reason or another). Now you need to rebuild something you’ve already spent time developing. Are you—and your boss—OK with that rework taking hours to days? Or do you think you’d be wishing you fell in the range of high IT performers, and you could quickly recreate what you lost in less than one hour?

Now, take it a step further. What could you be doing with that time you now need to dedicate to rebuilding something? What will that rework take you away from? The next phase of your development? Lunch? Happy hour? Sleep? What price would you put on the personal cost to you?

The 2016 State of DevOps Report authors wanted to calculate the cost of excess rework per year for organizations. They created a formula to calculate the number:

Cost of excess rework = Technical staff size x average salary x benefits multiplier x percentage of technical staff time spent on excess rework

Here’s what they found:
- High performers have 11 percent excess rework
- Medium performers have 22 percent excess rework
- Low performers have 17 percent excess rework
At first glance, those numbers seem odd. Why would the low performers have less excess rework than the medium performers? The report clarified that by explaining “low performers spend less time on rework … because they report spending a greater proportion of time on new work. We believe this greater amount of new work could be occurring at the expense of rework, thus racking up technical debt.”

It’s clear all organizations have some amount of rework as a result of a failure of some kind. The question is, What is the cost of that rework? Let’s take a look at the results of the study.

For the most part, the simple answer is that rework will likely cost your organization millions (unless it’s costing you billions). It’s just a matter of how many millions.

We often get asked why it’s necessary for OpenStack environments to have a data protection strategy and a backup solution. The conventional (but wrong) thinking is that virtualization can take the place of data recovery. Frankly, the answer to why is contained within those numbers above. If you don’t have a backup and recovery solution for your OpenStack development efforts, are you willing to spend money and resources on reworking the lost data? And if you are, how many millions are you willing to spend on it? Let me remind you that you are trying to recover to a point in -time … persistent data included. If you say, “If you built your cloud appropriately, this shouldn’t be an issue ….” Madame or Sir, if your virtualized cloud has that kind of resiliency, then I would like to work for you.
August 18, 2016
Backup and Recovery of MongoDB, Part 2

This blog is the second in a two-part series on backup and recovery for MongoDB. In one of my previous blogs, I covered why companies require both replication and backup for enterprise grade data protection. And in the first part of this blog series, I discussed the existing solutions for backup and recovery of MongoDB and their drawbacks. Now in this blog, I will discuss the key requirements for protecting data that resides on MongoDB (deployed either on-premises, on private cloud with an as-a-service model or in public cloud with Amazon AWS or Google Cloud Platform).

Requirement 1: Online Cluster-Consistent Backups

Next-generation applications deployed on MongoDB must be always-on. This means pausing the database to make backups is not feasible and the backup operation should not impact the performance of the application. As the application scales, the underlying MongoDB also needs to scale-out to multiple shards. In this case, a backup solution must provide a consistent backup copy across shards without disrupting database and application performance during the backup.

Requirement 2: Flexible Backup Options

Depending on the application, data may have different change rate and patterns. For example, in a product catalog, certain items may be refreshed every day (fast-selling goods, for example), while the others may have longer shelf life, such as premium items. Based on the application requirements, some collections may need to be backed up every hour versus others that may be backed up daily. Providing this flexibility to schedule backups at any interval and at collection level granularity is another requirement we have heard from customers using MongoDB. More importantly, these backups always should be stored on the secondary storage in native formats to avoid vendor lock-in.

Requirement 3: Scalable Recovery

During its life cycle, data resides in multiple stages such as development, test, preproduction and production, and also may reside in multiple clouds (private cloud and public cloud). The topology of MongoDB clusters at each stage is different. For production, the application could be deployed on a sharded MongoDB cluster on-premises, but the test team might have access only to unsharded MongoDB clusters in the Amazon AWS (public cloud). Hence, the backup solution should allow multiple restore operations such as sharded to sharded (such as from 5×3 cluster to 2×3 sharded cluster) or sharded to unsharded (such as 5×3 cluster to 1×3 unsharded) across such cloud configuration.

Requirement 4: Handling Failure

Failures are a norm in the distributed database world. However, the backup solution should be resilient to database process failures, node failures, network failure and even logical corruption of data during backup and recovery operations. Finally, the backup solution should be able to handle failures of MongoDB config servers that store metadata for sharded clusters.

Customers are deploying MongoDB in physical servers, private clouds and microservices such as frameworks, and in public cloud. Backup and recovery should be seamless across these deployment example and the ease of backup and recovery deployment is a big one for MongoDB customers and I will cover this in the next blog. Stay tuned!

July 12, 2016
Top 5 Reasons to Back Up Your OpenStack Environment
I am a big fan of lists. If you recall the movie “High Fidelity,” John Cusack’s character Rob and his co-workers—armed with an encyclopedic knowledge of all things musical—compile “top five” lists for every conceivable occasion, openly mock the tastes of people, and, every so often, sell a few records. In the spirit of a lists, here are the top five reasons to back up your OpenStack environment. We thought that it would be helpful to many.

If you’re currently working with OpenStack clouds, or are in the process of embracing it, you need to include backup and recovery for OpenStack on your list of to-dos from the start. You may ask, “Why? I have orchestrated/automated everything, I just need to spin resources up and down.” Cloud environments are fundamentally different from the virtualization we have become used to. Virtualization relies on infrastructure for resiliency. In the cloud, distributed applications are responsible for their own resiliency, and are independent of the underlying infrastructure. Therefore, cloud apps need backup and recovery options, lest you face a complete machine, data or file meltdown.

We’ve compiled the top five reasons to backup your OpenStack environment. Take a read and it will become clear why you must include backup in your OpenStack plans.
1. Compliance — Compliance may seem like an issue for only certain industries—finance and health care come to mind—but nearly every organization has to deal with compliance issues. For some, it’s compliance to external regulations, while for others it’s simply complying with internal policies and best practices. Backup for OpenStack is an essential component for compliance; it ensures the availability of the data at a specific point in time within your OpenStack applications. I was recently involved in a dialogue/string on LinkedIn watching experienced OpenStackers go back and forth debating “pet vs. cattle.” “I am getting tired of it,” one person said. “This is cloud. It’s about scaling up and out … With automation and orchestration there is no need for cattle.” This conversation went on for days. Theory is great. Practice is reality. What people failed to discuss is that compliance or the business units will drive many requirements (such as backup) either today or tomorrow. End of story.
2. DevOps — Backup is often the forgotten tool for DevOps. Containers are great to use, but the ability to test with real data is an issue. The reason to use backup is quite simple: test and development. With a copy of a production environment, you now can test upgrades and new features in your agile development process. Furthermore, if you lose a machine or your data, or you end up with corrupted files while in the middle of testing or development, your life will be much easier if you have an appropriate backup solution in place. Restoring machines, data and files is a better option than re-creating them entirely. Cookbooks and recipes are great as well, but you still need to couple it with the appropriate “ingredients.” It is like playing with a fake kitchen as a child … do you want to make breakfast with plastic pancakes and toast? Nothing beats the real thing. Yes, more resources are required for one over the other, but when something goes wrong—it is unfortunate—people point fingers.
3. Disaster recovery plans — While the terms “backup” and “disaster recovery” often are used interchangeably, they are actually not the same thing. You might have what appears to be a great recovery environment (ideally one that mirrors your production environment for high availability), but without a comprehensive backup solution, your recovery might not be the painless experience you expect it to be. Perhaps the solution for intelligent disaster recovery is to marry backup with a Swift environment so you can recover from any point in time, anywhere in your organization.
4. Save time — Often recovery can be a frantic process, driven by a business need to have the machine, data or files recovered as soon as possible. Or, more likely, sooner than possible. If you are trying to re-create something under that type of environment, there’s a likelihood that mistakes could be made. This becomes increasingly true the more complex and mature your cloud becomes (think about the growing number of workloads and applications). A much easier approach—one that provides you with peace of mind—is to have a comprehensive backup solution ready and available for when you need it. There’s also another reason backup can save you time: When you have to re-create the lost information, you are taking your time away from something else—and that means you are taking a big hit to your productivity. Save yourself and your organization a big headache with backup.
5. Security and forensics — You never know when you are going to be asked to provide data or files related to a security issue or a forensics investigation. The right backup solution not only makes it easier to assist with the activity, but think how good you’ll look by being able to deliver the information the forensics team needs quickly.
Are you currently looking into backup solutions for OpenStack? Now that you know why you need backup for OpenStack, learn more about how to backup OpenStack.
March 10, 2016
Multi-Tiered Escalation Policies are Paramount
As any team charged with keeping systems up and running will tell you, having one person on-call isn’t enough. What happens if your on-call engineer sleeps through his alert? What happens if his phone’s battery dies without him knowing, or if he gets an alert at an inconvenient time, like when stuck on a bus or in traffic? It will happen.

In today’s always-on world, companies can’t afford for that to happen without a backup plan. One or more people should always be waiting in the wings, ready to spring into action if a primary on-call is unable to perform his or her duties to the best of his or her abilities at any given time.

Engineers who perform backup duty don’t need to be as “on-call” as a primary engineer — no pagers needed here — but what backup engineers lack in readiness they can make up for in numbers. It often makes sense to have multiple backups or an entire team of backups.

Operationally mature companies often group the currently-on-call engineer and all of his or her backups into an escalation policy. Escalation policies set the order in which each person is alerted and the delays between alerts. While there are many ways to organize these escalation policies, here are some of the patterns most often used:

Primaries Aren’t Enough; Determine a Secondary, Too
Most companies have a primary on-call engineer, usually determined by a calendar that designates when members of an ops team are on call.

Operationally mature companies take their on-call practice one step further and supplement this rotation with a rotation of secondary on-call engineers. Why a secondary rotation instead of just paging the whole team when the primary falls through? If a large team is paged, you get a ‘tragedy of the commons’ effect— it’s urgent that somebody respond to the issue, but not clear how urgent the issue is that each individual person respond. Having a designated secondary removes that confusion.

The secondary rotation is set up to shadow the primary rotation and generally has the same members as the primary rotation. The difference is that the order of the engineers in the secondary rotation is usually offset from that of primary rotation— if your primary and secondary are the same person, you’re going to have a bad time. For example, if a primary rotation contains Alex, Bob, then Charlie, then a secondary rotation may contain Bob, Charlie, then Alex in that order. Companies often find that if last week’s primary is this week’s secondary, the it helps make sure there’s somebody with recent context who can help if the primary runs into trouble. It is becoming more commonplace for organizations to have both a primary and secondary on-call rotation; but, in an environment where customers expect zero downtime, secondary rotations are no longer an optional tool.

Enable a Strong Support System
Larger companies and companies with significant numbers of operations tasks often deploy a separate support team that handles all basic operations tasks before the organization elevates the tasks to an on-call engineer. This team — the first tier support team — usually knows how to handle all of the smaller, repetitive problems that crop up often and have clear resolution procedures. These first-tier teams often offer their support to multiple engineering teams and are placed first in an escalation policy.

If first-tier teams cannot resolve the issue, escalation policies typically dictate that primary then secondary on-call engineers must complete the task. However, there is one more support network operationally mature organizations must implement: management. Management is ultimately responsible for a company’s systems, and needs to be in the know when severe problems happen— both to hold their teams accountable and to muster additional support. At PagerDuty many of our EPs have a rotating manager schedule at the top of multiple escalation policies, and many of our customers use this pattern as well.

Put Your Tiered Teams to Work
In practice, an organization policy for a hypothetical “Database Ops” team might look like this:
- Assign the incident to the user who is on-call in the First-Tier Ops Team schedule
- Assign the incident to the user who is on-call in the Primary DBA schedule
- Assign the incident to the user who is on-call in the Secondary DBA schedule
- Assign the incident to the team lead
- Assign the incident to the dev manager
This escalation policy would unfold over time, with timeouts of between 10 and 30 minutes before each subsequent escalation, depending on a team’s needs.

Implementing an escalation policy that involves multiple tiers of resolution ensures that devops teams address all alerts — whether business-critical or minor — in a timely manner. Limited downtime is paramount in today’s fast-pace business environment. Customers expect their applications to run smoothly, or they expect to switch providers if devops teams cannot resolve issues fast enough. Organizations will be much less likely to see their processes fail, and much more likely to see happy customers, if they implement a multi-tiered escalation policy.
December 11, 2015