devopsy.kinsta.cloud

Tag: continuous incident response

  • Do You Need Your Ticketing System for Real-Time Incident Management?

    Do You Need Your Ticketing System for Real-Time Incident Management?

    The term “operationally mature” characterizes companies that understand how incident management affects both employee well-being and business impacts. Operationally mature companies have a process in place with built-in reliability and accountability that limits high-severity incidents because of bugs. The more mature a company’s operations, the better it can maintain agility and adapt to change. So how should operationally mature organizations handle incident management? Are ticketing systems still the gold standard?

    Ticketing systems keep track of issues, assign items to other people and order requests. They are a great way to manage a queue of ongoing requests, yet many operationally mature companies shy away from ticketing systems for their real-time incident management.

    Likely, it’s because ticketing systems can be fraught with issues. A clunky workflow, mired in process, means that users can’t always move and adapt quickly. Every minute they spend updating their workflow in the ticketing system equates to less time for dealing with critical issues. Ticketing systems often require a rigorous, structured, manual approach that can slow down work in the heat of the moment.

    Leading DevOps companies are proving that you don’t need a process-heavy system to get work done. In fact, for real-time incident management, the less process, the better. Companies are migrating from a system where a user manually opens a ticket when they see an outage, to monitoring systems that automatically generate alerts as problems arise. As soon as something goes wrong, the system generates an alert. There is no delay waiting for someone to notice the issue and fill out a form. Teams get immediate visibility into what’s going on with the system and can begin work or escalate accordingly.

    Operationally mature companies really need a lighter-weight solution. They need a solution that focuses on automation, allows them to be more agile and fixes incidents faster. The decision is not an either/or proposition, however.

    Many companies still have a place for ticketing systems, even if they aren’t for real-time incident management. Organizations can combine ticketing systems with more agile solutions. A more lightweight system still allows users to open manual alerts if necessary, much like a ticketing system does. Some operationally mature companies use a ticketing system to track incident follow-up and generate work estimates for lower priority tasks. Once the critical incident is resolved and service is restored, companies can create tickets to fix the root causes and make sure the problem doesn’t happen again.

    Ticketing systems are not necessarily outdated or obsolete. Unfortunately, one of the benefits of a ticket system is also a weakness. The plethora of fields, for example, allows users to track various aspects of an issue, but those fields are cumbersome and unnecessary for high-priority items. Ticketing systems can be clunky when DevOps needs something sleek. Combining ticketing with a lightweight, automated, incident management system is a sure-fire way to streamline some of the extra processes.

    Smart companies ask, “How much process is necessary to accomplish our tasks?” Answer that question, and your IT team will thank you. The answer will help them feel less like a help desk and more like an integral member of the DevOps team. And that helps your organization move faster and get closer to achieving DevOps Nirvana.

  • When the bad guys use DevOps, you need DevOps to defend yourself

    When the bad guys use DevOps, you need DevOps to defend yourself

    I came across an intriguing headline the other day: Malware coders adopt DevOps to target smut sites. The article, unfortunately, is fairly vague about what exactly that means, but it got me thinking about the potential implications.

    It’s all just a function of rivals competing. Just as Company A has to find a way to work more efficiently and bring better products and services to market faster than Company B, malware developers need to be faster and more agile than their targets. In either scenario DevOps can give one or both parties a competitive boost.

    Although the article in question simply says malware developers are “now engaged in DevOps,” and claims it’s the first time that behavior has been seen from the dark side, the reality is that malware developers—even those in organized cybercrime groups—have always had an element of DevOps to them. By their very nature they’re more inclined to have shared responsibilities and to exhibit the sort of jack-of-all-trades cross-functionality that we expect to find in young startups.

    “It has always been the case that most attackers embrace much more of a DevOps pattern than defenders. Most of these folks were born of the Internet way of thinking and nowhere in their playbook do they have the traditional data center change controls or any other big enterprise methodology,” declared TK Keanini, CTO of Lancope.

    The problem that many organizations face when it comes to mounting an effective defense is a lack of situational awareness of their own environment. In many cases the attackers actually know more about the organization’s network, and the assets connected to it than the organization itself.

    Effective defense also requires agility. Companies need to be able to adapt quickly and implement changes without impacting business continuity. Keanini says, “DevOps is one way to do this and something like the Rugged Manifesto does a great job of capturing the essence of what it will take to battle this threat day in and day out.”:

    The Rugged Manifesto

    I am rugged and, more importantly, my code is rugged.

    I recognize that software has become a foundation of our modern world.

    I recognize the awesome responsibility that comes with this foundational role.

    I recognize that my code will be used in ways I cannot anticipate, in ways it was not designed, and for longer than it was ever intended.

    I recognize that my code will be attacked by talented and persistent adversaries who threaten our physical, economic and national security.

    I recognize these things – and I choose to be rugged.

    I am rugged because I refuse to be a source of vulnerability or weakness.

    I am rugged because I assure my code will support its mission.

    I am rugged because my code can face these challenges and persist in spite of them.

    I am rugged, not because it is easy, but because it is necessary and I am up for the challenge.

    The question is, “How?”

    Keanini—ever the strategist—references the OODA loop, conceived by USAF Colonel John Boyd. OODA is an acronym for Observe, Orient, Decide, and Act. The concept was originally designed for military combat operations but can also be applied quite effectively in many areas of business. Applying it to network or endpoint security, the idea is to at least raise the cost of observation and orientation for the opponent—in this case the malware developers.

    “DevOps can deliver this because change can be implemented as a defense whereby the adversary never has time enough to make enough of an observation, or orientation accurate enough for effective decision and action,” clarified Keanini.

    There are a few different elements involved here. Continuous monitoring and continuous incident response are both applications of a DevOps mentality that can help organizations mount a more effective defense against sophisticated attacks. The OODA loop Keanini refers to is less about response, and more about applying DevOps philosophies to take away the strategic initiative from the attackers in the first place.

    One thing is certain, if the malware developers implement policies that enable them to craft and deploy attacks faster, businesses will have no choice but to figure out some way to adapt and defend more quickly as well.

  • Embracing a culture of continuous incident response

    Embracing a culture of continuous incident response

    In a world of continuous development and rapid iteration combined with being connected online 24/7/365, you are bound to encounter threats. In fact, attackers use automated tools so you are under constant siege from continuous threats. The natural response is to implement some form of continuous incident response.

    The standard development model involves different stages of requirements gathering, writing code, and beta testing, resulting eventually in a defined, periodic software release. That model is being replaced by the DevOps culture of continuous development. Security faces a similar transformation from the pervasive legacy model of the “us vs. them” protect-the-perimeter model to a new system of continuous incident response that addresses the current threat landscape in the real world.

    There is no “us”, and there is no “them”. There is no perimeter. Your authorized users are actually more likely to expose sensitive data, or compromise your systems—whether intentionally or inadvertently—than external bad guys. Your authorized users are also just as likely to be “outside” of the mythic perimeter—connecting from personal smartphones or tablets, or logging in from a Starbucks across town. The traditional security model is simply not equipped to deal with today’s technologies or threats.

    So, what is continuous incident response? I spoke with TK Keanini, CTO of Lancope, who explained, “Some people when thinking about process think of it as a linear model where there is a beginning and end; a better way to model adaptive processes is with a loop and continuous incident response is a loop.”

    He elaborated, “Borrowing from John Boyd’s OODA loop, continuous incident response is where there is an observation, orientation, decision and action cycle that outpaces or runs at a faster tempo than the adversaries.”

    Zane Lackey, director of security engineering for Etsy.com, gave a presentation at the Hacktivity conference on the topic of “Attack-driven Defense”. The concepts he described are essentially the sort of logic and pattern that need to be applied for continuous incident response.

    There are a few key elements organizations need to embrace in order to effectively implement continuous incident response. The first is the loop Keanini described. In order to be “continuous”, you have to continue to cycle through the various phases. There is no “end”.

    One of the crucial components of continuous incident response is continuous monitoring. It isn’t good enough to scan the network once a month, or once a week, or even daily. The threat landscape is constantly changing and evolving, and you have to be continuously monitoring for new vulnerabilities, new attack techniques, and anomalous or suspicious activity that might indicate a security incident is occurring.

    Another important factor is to view continuous incident response as a cross-functional business process. Keanini stated, “I think continuous incident response—when implemented well—touches all critical parts of the business, not just IT. Making it an IT thing is just too narrow and people who experiment in this manner are destined to fail.”

    Companies that have a DevOps mindset, or have adopted DevOps principles will find it easier to relate to the concept of continuous incident response. DevOps organizations are not constricted by the departmental barriers of traditional companies. The hierarchy and red tape of an org chart can get in the way of effective continuous incident response, but DevOps businesses have a more open, fluid organizational structure that makes it much easier to adapt to the dynamics of both operating on, and responding to threats from the Internet.

    Adopting continuous incident response may be easier said than done. It is a different approach to security, which may make it more difficult to get buy-in and support from executive management. You may also encounter resistance from the IT admin and existing security personnel because they’re used to doing things the old-fashioned way, and change is uncomfortable.

    Keanini sums it up well, though. “Just step back and ask yourself how important being connected to the Internet is to your business or customers—and then just consider that continuous incident response is the cost of doing business on the Internet.”