devopsy.kinsta.cloud

Tag: failure

  • DevOps Future: Have a Break Plan

    DevOps Future: Have a Break Plan

    A big tenet of DevOps is, “Automate all of the things.” It’s generally a good policy, and definitely allows us to meet the delivery goals of DevOps. Taking the full-on, all-new technology case, an org could run full automation on the dev side and deploying with an application release automation tool to a fully automated container management system—fast, efficient, programmable, even portable. This level of automation absolutely enables DevOps.

    Until things break.

    Ignore the well-known and easily overcome cases such as, “That container is no longer responding,” and focus on the larger infrastructure environment. Do you have a plan for getting builds out if DevOps has taken you to many builds a day and your entire Jenkins environment just crashed? The easy answer is, “Switch to another Jenkins server,” but if you are big enough to have more than one, workload may be an issue, as might be the Jenkins environment itself. The usual example of Jenkins for client builds and separate Jenkins environment for server builds is a good example. If the client environment drops, you can’t just move it to the server build environment. Test tools and other tools for clients will have to be moved over, and almost certainly the build environment will have to be modified.

    Along those same lines: What if, instead of a few containers dropping, your container management system fails? (It’s software, it will happen.) What’s the plan then? Depending on the type of failure, it could take your entire application or application portfolio down with it. In most organizations, downtime is not something taken lightly, so what is the plan to return to normal function? Waiting until you’re in the middle of a disaster caused by a single bad disk, failed memory chip or an attack exploiting the infrastructure is a bad plan. We’ve been doing this for decades, yet we know software fails, and sometimes it fails spectacularly.

    Once the first rush of DevOps is through, processes are streamlined and automation is basically in place, it’s fair to take a deep breath and plan for how to overcome the break and total failure of each tool in your toolchain. Yes, those tools will change and you’ll have to do a little work each time you add/change one, but what is your preference: a slightly slower adoption of DevOps after the low-hanging fruit is gathered or confront a failure of infrastructure proportions without a plan?

    Long before computers, Benjamin Franklin is credited with having said, “By failing to prepare, you are preparing to fail.” That credo has been repeated often since. There is truth in that statement with any form of automation. That which is automated eventually goes awry. You can plan for it, or suffer the stress of trying to figure out what to do while it’s an emergency. I suggest having a break plan.

    — Don Macvittie

  • In Determining DevOps issues, Sometimes It’s the Process

    In Determining DevOps issues, Sometimes It’s the Process

    It is pretty much standard DevOps process that when a server instance starts having problems, you simply kill it and start another. This is in line with the idea that servers are cattle and there isn’t a ton of difference between them.

    But it creates a problem that no amount of CI/CD or automated provisioning can overcome. That’s the blindness problem. CI and CD miss some bugs, simply because of the wild variations possible in inputs when humans are involved, or unique issues of hardware or platform that DevOps tries to ignore as much as possible. While our tooling cannot be so over-arching that we have data points on everything, and we can track down problems to the routine that is causing them, a service degrading performance or outright crashing is indicative of one of those blind spots.

    If you’ve been in a highly dynamic DevOps environment, you know this is no simple problem. But it is one we have to resolve, because killing and restarting is simply masking a problem. Indeed, it is this very process that turned Cloudbleed from a simple and understandable programmer error into front page news. They knew their servers would occasionally have memory over-runs, even crashing on occasion; they would just spawn another instance and keep moving along. The ability to create another so easily reduced their desire/need to fix the underlying problem. But the problem was bigger than they knew.

    The problem we face is, of course, how much time and work is invested in monitoring, and how do we know what’s really important. This problem gets worse when you have an instance that stops responding, or starts hogging resources. Getting it offline is imperative, but re-creating the issues that caused it to go off the rails may not be so easy. In fact, often it is not at all easy.

    Long term, we need a way to pick up the exact problem area, create a bug report and get it back into the Dev/CI/CD system that is reliable. Short term, don’t ignore errors in production unless you know exactly what is going wrong.

    Sure, it’s easy to kill and/or restart an instance to “recover” from an error. It’s easy to chuck your app on the public internet without any security, too. Just because it’s easy doesn’t make it a good idea.

    So watch those logs, don’t mask problems and stay on top of it all, even if the complexity sometimes has you reeling a bit. Because application reliability might be impacted—and application availability is, in the end, the whole point.

    — Don Macvittie

  • Best DevOps Practices for 2017

    Best DevOps Practices for 2017

    Because the idea of DevOps is still so new, we’re continually having to hone what we view as best practices. At the start of the new year, many people make their own personal goals clear for the year (regardless of whether they intend to keep them), but we tend to head back to our work desks without reassessing how our projects are going. We just jump back in unquestioningly and continue down the path we were on before the holiday.

    Mindless plodding away at a goal that had been set too far in the past is perhaps the biggest business mistake we can make. Re-evaluating our approach is often the difference between stagnation and real innovation. A willingness to change tacks and the agility to do so speaks volumes about an organization—that flexibility is rewarded by a market that demands responsiveness from its providers, and it can be a big differentiator between a company and its competition. With that in mind, there are six big goals that we can adopt as DevOps practitioners that will have enormous results on our ability to answer the demands of our users.

    Make paying off debt a priority

    Most companies are so saddled with technical debt that they hire more people just to cope with it. This is like putting more hoses on a forest fire. When fighting a forest fire, a portion of the team creates a firebreak to deprive the fire of fuel. Likewise, organizations need to dedicate a portion of their workforce to do nothing but eliminate the technical debt that is fueling their operational fires.

    Set ambitious goals, but don’t sweat the details

    Have goals and objectives, but allow the minutiae to be developed over the course of short intervals. We need to set business-level objectives for our DevOps teams, but we have to be careful to not overprescribe the details. There are many DevOps tools and methods and more arrive every day. Plan to work in two- to four-week sprints with measurable outcomes, instead of six-month project plans. Use the feedback and lessons learned from those short sprints to inform the next sprint’s work.

    Don’t leap, but get moving

    DevOps is meant to be about quickly moving product from left to right. A common mistake is for companies to wait for “the perfect moment” and then jump headlong into DevOps, creating a lot of disruption at once. This is a recipe for disaster. Identify workstreams that are ripe for improvement from DevOps and start by moving one new benefit from concept to production. Remember, single-piece flow is your friend.

    Identify the ‘credit hawks’ and neutralize them

    I once heard the saying, “Just because you run out in front of a parade doesn’t mean they are actually following you.” One of the critical pieces that makes DevOps work is a culture of sharing. That means sharing in knowledge, work, credit and blame. A “credit hawk” is someone who likes to jump in front of work already being done and claim he or she is orchestrating it. This seems to be commonplace in situations where DevOps teams exist under mid-to-upper managers with “old guard” mentality. If they take the credit, that leaves the DevOps team with the blame. Neutralize the credit hawks at all cost; they will undermine your culture.

    Plan for failure, success through learning

    Planning for failure does not equate to “expect failure,” but because we want to move faster so we can try more things, the law of averages tell us that we will also fail more, too. Planning to fail means we know how to conduct a blameless post-mortem, understand lessons learned and incorporate that feedback cycle to increase the likelihood of success.

    Promise to break the rules

    Your DevOps team needs to be trusted to find the best path, even if that means ignoring a 70-year-old process. This statement might invoke a knee-jerk reaction, but this does not imply “anything goes.” Your objective is to find more productive ways to get product to market and to create safe systems of work. Part of your lease to your DevOps teams also should be to explore new processes that eliminate red tape, bureaucracy and waste. What has worked for the last 70 years may be what is holding you back today.

    2016 rewarded those who were able to move fast, and 2017 promises to require us to move even faster. By resolving to make these changes in your company or organization, you are making a commitment to shed the shackles of yesteryear. When freed of these fetters, your application teams will be able to focus and deliver on the one thing we are all trying to do: deliver value directly into the hands of our customers. Is 2017 the year your company will triple its release cadence? Is this the year you deliver every new product into all markets simultaneously? Or perhaps this is the year when your applications suffer zero outages. The goals and the stories are yours to tell, but one thing is for sure: Resolutions without action are just wishes.

    About the Author / Adam Bowen

    Adam Bowen is an information technology leader with more than 15 years of experience in technical, sales and senior leadership roles. As the Worldwide Innovation Lead at Delphix, Adam works with thought leaders of the Global Fortune 100 to eliminate key business constraints through innovative new  processes, offerings, and technologies. Connect with him on Twitter and LinkedIn.

  • Is Your Company’s IT a Disaster Waiting to Happen?

    Is Your Company’s IT a Disaster Waiting to Happen?

    Recent IT disasters suggest gaps in hardware testing, backup system testing and inadequate disaster recovery plans.

    Although the summer often is the most profitable season for most airlines, this past summer wasn’t great for several of the largest carriers. In July, Southwest Airlines suffered a 12-hour IT outage, triggered by a router malfunction that quickly took down many of its core operating systems like a house of cards. Worse, backup systems failed to deploy. Flight delays and cancellations continued for a few days.

    A similar disaster unfolded at Delta in August, when a power control module at the airline’s central data center malfunctioned, creating a surge to the transformer and loss of power. Delta’s systems didn’t switch over to backups as planned, causing scheduling mayhem as agents couldn’t access key systems. For both companies, the outages resulted in tens of millions of dollars in lost revenues.

    Sure, both of these outages affected businesses running more traditional IT infrastructure, but even if your company is running in the cloud, your business isn’t immune from a damaging outage. In September, Microsoft Azure experienced two outages lasting several hours each, one in Europe and the other affecting U.S. customers across several key services, including SQL Database, Virtual Machines, Service Bus and Visual Studio team services.

    As software developers and testers, we often focus narrowly on the application and the server it is running on—but there’s a lot more to consider. Hardware and components, backup systems and third-party providers all come into play when testing applications for stability and reliability. Of course, there’s no way to prevent all bad things from happening but there are some steps you can take to cover the bases more thoroughly.

    Look Beyond the Software

    If there were three main lessons learned from these recent IT disasters, they would be:

    1. Assess risk of the whole environment. The quality of your software is only as good as the components it runs on—the hardware, network, power, storage, and other third-party applications are often required. Understand the full picture and all of the dependencies. Categorize applications based upon business criticality and failure rate to focus attention on the pieces that really matter.
    1. Backup systems and processes are not “set and forget.” Spend more time documenting them, testing them and training staff. Make sure that the processes are up to date as IT adds new components and vendors to the system or if the company goes through a merger or acquisition.
    1. Invest according to your risk profile. If your company can’t operate for more than a few minutes without access to certain systems, plan on budgeting for warm or hot failover systems which enable near 100 percent uptime with always-on, parallel systems continually synchronized and ready to go. It’s too expensive to enable this failover capability for your entire environment—you will need to be picky. Some IT assets, such as an airline’s scheduling and reservation systems, deserve top priority and resources, while other assets such as the HR system, can do with less. IT directors also need to frequently investigate the need for upgrading old, unstable infrastructure technologies. Most companies also don’t spend enough time monitoring and testing backup and recovery systems.

    To minimize the risk of a major IT outage bringing down your business and alienating customers, the recipe is as follows: Analyze the points of failure, determine what is reparable and what is not easily fixed (for example, installing software usually can be done much more quickly than building a new server environment) and, finally, install and test failover systems and processes that can maintain basic operations with minimal customer disruption.

    Delta & Southwest: A Quick Analysis

    In both cases, Delta and Southwest had backup systems that did not deploy properly—or, at least, in a timely fashion. The first point of action would include conducting thorough, regular testing of those backup systems and locations and upgrading capabilities if necessary. Second, since no disaster recovery system is foolproof, create manual workarounds for critical system outages. That would have helped tremendously so that customers weren’t completely stranded. Third, create a fully redundant second data center that switches over automatically when the primary data center goes down. Naturally, this is the most expensive option, but depending upon the outcome of the risk analysis, it may be worth the money.

    With Southwest, a router failure caused a cascading effect that brought down the network altogether. It’s not clear if a secondary network was available for failover, although it appears the answer is no. Again, better, more frequent monitoring could have isolated the router problem and alerted IT to fix or replace the hardware before disaster struck.

    Any company with technology from many different vendors should work closely with vendors and partners whenever possible to mitigate possible risks. Vendors often are able to provide insight into how they test their own systems and can even share test cases to assist in validating the end-to-end system. Keep in mind that aging technology is not the only problem—sometimes new technology brings untenable risks. A recent example is the Samsung Galaxy Note 7 and its overheating battery problem and recall. As well, it’s smart to understand how the local utility handles its outages and what if any help it provides to customers in the event of service disruption.

    Planning Ahead

    Large companies with complex IT infrastructures should take a holistic view of critical system availability, considering all of the possible ripple effects from a single point of failure. Today’s systems and networks are increasingly interconnected and interdependent. For both airlines, the effect on customers and employees lasted far beyond the actual outage, as managers struggled to get agents and planes reorganized and rescheduled. Take that risk into any spending calculations for the disaster recovery plan. Companies that have gone through mergers and acquisitions have additional complexities from IT integrations—or lack thereof. Failover strategies should ideally be incorporated upfront as part of any acquisition plans.

    Comprehensive testing strategies are paramount, although sometimes overlooked, in preventing outages. Request information from core vendors, such as the networking equipment provider, on testing processes and ask to see their test cases. Though the vendor should be responsible for ensuring thorough testing of their own products, the customer is always the one to suffer the costs of a downed system. Development and testing teams should get clear on how external systems and providers handle down time and equipment failure. Strive to align the company’s disaster recovery processes with those of partners.

    Finally, for companies running mission-critical systems in the cloud, there are ways to mitigate the effects of an outage. Multi-cloud management tools such as CliQr (now owned by Cisco) allow IT to quickly redeploy workloads from one cloud service to another whenever needed. Containers are also helping in this regard due to the inherent portability benefits of the technology. Companies are using containers to move applications and workloads from region to region within one cloud provider; from cloud to cloud, such as from AWS to Azure; and even from the cloud to an on-premise data center.

    There are many causes of IT outages today, and as IT environments become more multifaceted, supporting more technologies, data and applications all the time, those risks grow. We don’t have an answer to the problem, but there is one strategy: leave no stone unturned. Involving testers and developers in outage prevention and disaster recovery planning is a wise move, indeed.

    About the Author / Kevin Dunne

    Kevin Dunne picKevin Dunne is vice president of strategy and business development at QASymphony. Prior to joining QASymphony, Dunne was a business technology analyst at Deloitte. He can be found online at Twitter and LinkedIn.