devopsy.kinsta.cloud

Tag: observability

  • The Hybrid Cloud and Why DevOps Teams Need Unified Stack Observability

    The Hybrid Cloud and Why DevOps Teams Need Unified Stack Observability

    The modern data center is constantly evolving to keep up with the demands of always-on, anywhere computing. Long gone are the days of monolithic, on-premises data centers. Today, the computing modus operandi is that of the hybrid cloud, which, according to Statista, will be worth close to $50 billion in 2018—just about double what it was in 2014—and will grow to almost $92 billion by 2021.

    As more and more DevOps and security teams are integrating hybrid cloud environments and serverless architectures, they need to gain visibility and insight into the increasingly distributed workloads provisioning and supporting their applications. Without a single pane of glass across their hybrid clouds, Dev, Sec and Ops teams cannot quickly discover vulnerabilities or identify and respond to attacks.

    The emerging best practice to the cross-cloud visibility conundrum is unified stack observability. Unified observability is the ability for Dev, Sec and Ops to share the same application-aware monitoring environment for vulnerabilities, threats or attacks across all workloads supporting their applications. The idea behind unified observability is that each team can look at the different stack layers as one—with an app-aware, single-pane, top-down view. For Dev teams, unified observability means that they can see and define all the third-party services connected to their applications; for Ops, it means they can see and monitor all workloads. And for Sec teams, it means the ability to set security policies for detecting and blocking attacks at each layer of the app stack by enforcing policies uniformly. There are a few recent trends accelerating the need for unified stack observability.

    The Accelerated Need for Unified Stack Observability

    Recent computing and application development trends are changing the way Dev, Sec and Ops teams develop and manage their distributed applications across hybrid cloud infrastructures. These changes include the increased use of third-party services, an overabundance of purpose-built security tools and the increasing need to address the shared cloud responsibility model, among others.

    An Uptick in Third-Party Code Use

    Third-party code increasingly is being used in apps and is one of the critical attack vectors exploited by hackers. SafeCode research notes that 78 percent of companies are running part or all of its operations on open source software, while 66 percent create software for customers built with open source. These stats have nearly doubled since 2010. The very practice of code-lifting, or pulling open source code from different repositories, which has enabled developers to share innovation and collaborate across architectures and levels, is turning to be one of the biggest attack vectors for hackers to exploit. Running on many different functions, open source applications and code borrowed from third-party services increasingly are becoming more vulnerable and therefore require a need for constant observability and monitoring.

    Even though such code is usually vigorously tested for vulnerabilities during the build and deploy stages, it can—and in some cases is—compromised in production. The global Magecart credit card skimming incident is an indicator of this: The hackers were able to infiltrate the third-party components and exploited them to gain credit card information, resulting in 800 e-commerce sites being compromised. To avoid such distributed vulnerabilities, Dev, Sec and Ops teams need to constantly index third-party functions and services, and scan for possible vulnerabilities or attacks. Such monitoring requires unified observability at the app code level.

    The Growing Overabundance of Purpose-Built Security Tools

    Increasingly enterprises are leveraging multiple cloud providers while still running their own data centers, either on-premises or as private clouds. A 2017 report from Forrester Research points to 89 percent of enterprises using at least two clouds, while 74 percent use three or more public clouds. Despite a wide industry effort lead by global leaders such as VMware to provide seamless cross-cloud orchestration and management, most clouds remain in their own environments with a set of purpose-built management and security tools.

    Then there is the traditional security approach, which can get in the way. According to Cybersecurity Insider’s “2018 Cloud Security Report,” only 16 percent of organizations believe the capabilities of traditional security tools are sufficient to manage security across the cloud. Eighty-four percent said traditional security solutions either don’t work at all in cloud environments or have only limited functionality, meaning that more tools with specific functionality must be integrated within the cloud. On-premises, private and public clouds remain siloed for most team, stressing an overtaxed security resources across organizations.

    Shared Responsibility Can Be a Shared Burden

    Even though the shared responsibility model has been well-documented, securing public cloud environments can be a burden to security and IT teams as their responsibilities continue to grow, while available talent is short to find. The traditional shared responsibility model assumes that the cloud vendor secures the hardware and software of the cloud itself, while the customer is responsible for the security of their assets within the cloud.

    With the shared responsibility model, most security teams are unsure about how to do their part. Gartner reports that through 2022, at least 95 percent of cloud security failures will be the customer’s fault. This can include anything from a failure to monitor to a failure to adequately secure the correct elements. This can also include customer misconfiguration, mismanaged credentials or insider theft. Additionally,  The “2017 IBM X-Force Report” noted a 424 percent jump in breaches related to misconfigured cloud infrastructure, largely due to human error. These gaps in security leave security teams with their organization, data and apps vulnerable.

    App-Aware Stack Observability: The New Modern Security Mandate

    The biggest distributed security blind spot facing modern Dev, Sec and Ops teams today is for them to monitor for vulnerabilities or attacks at each layer of the stack—at the application/third-party code-use level, their distributed workloads and the network each in isolation. Unified application-aware stack visibility and observability is becoming the only way to fully see and understand potential risks in today’s increasingly distributed environments, effectively addressing some of the biggest security challenges faced by modern Dev, Sec and Ops teams.

    — Gadi Naor

  • A Vendor Guy Goes To Monitorama

    A Vendor Guy Goes To Monitorama

    I attended my very first Monitorama this year, in Amsterdam, and I have thoughts!
    Monitorama Amsterdam
    Monitorama Amsterdam

    First of all, it’s deeply weird speaking at Monitorama when you work for a vendor in the monitoring and observability space, but you’re not there as a sponsor. This would be pretty odd at the best of times, but the crowd at Monitorama are next-level. They are very comfortable with highly technical content, so they get irritated very quickly with (what they perceive as) marketing fluff. “Down in the weeds” doesn’t even begin to describe it; most people had never heard of the sorts of things that are part of life in enterprise IT (AIOps, Netcool, Patrol or any of our usual suspects), but, on the other hand, everyone I met could engage in a multi-hour argument over which monitoring tool is worst.

    In fact, I would say that one of the highlights of Monitorama was the conversations in the hallway track. This is where all the Twitter loudmouths—I mean, “influencers” and “thought leaders”—who drive this industry congregate. If you manage to get into a conversation with them over coffee or beer, both of which were flowing freely, there can be a real exchange of ideas. I know I learned probably as much from those small-group chats as from the presentations on the big stage. I hope I was able to share some perspective in return.

    Heck, as ridiculous as it sounds, even the dress code was a challenge. At the events I attend as a vendor, there are basically two speeds: full suit-and-tie battle dress or a T-shirt with my employer’s logo. Neither case requires much thought when packing. Monitorama, though? I’m no longer credible in my old Thinkgeek shirts from my sysadmin days, but I’ll be laughed out of the room if I show up in a tie.

    Preparation

    I had answered the Call for Papers with a pitch about AI for distributed systems. Here’s my abstract:

    The IT industry is in the middle of one of its regular swings between centralization and decentralization, driven by increased automation of the network fabric itself, as well as new use cases such as IoT. With more and more processing and autonomy devolved to the edges, old assumptions about how to manage and operate a network have to change. It no longer makes sense to try to forward all the edge alerts to a central location for analysis, but central visibility on the health of the network is more important than ever.

    New techniques in AI-enabled observability hold the promise of helping NOC teams deliver better experiences for users of their networks, without requiring excessive manual work or heavy impact on on Operations teams’ personal lives. Machine learning enables knowledge to be captured and made available in context when it is most useful, accelerating incident resolution and improving user experiences.

    My first hurdle was that, although I have public-speaking engagements at least once or twice a month—and have done for some years now—I normally appear under the flag of my employer. This fact has both pros and cons. On the positive side, I can usually rely on a substantial library of slides that I (and others) have created in the past. I have some known high notes that I will be aiming to hit, and the call to action at the end is pretty explicit (“try our stuff”). The other side of the coin is that, because the audience is aware of these factors at least at some level, I have to work that much harder to earn their attention, as they assume they are being sold to. As a vendor spokesperson, I am also somewhat constrained in how wacky and off-the-wall I can get—although I do always try to push the boundaries. At one speech last year, I talked about bicycles for several minutes, to the consternation of the organizers, until they worked out what I was building up to.

    So I slaved away for some time over a hot Keynote, watched videos from past Monitoramas, consulted people in the industry whose opinions I respect and rocked up in Amsterdam pretty pleased with myself—to the point that I stayed out at In de Wildeman perhaps later than I should have on the night before Monitorama kicked off.

    Not Enough Preparation

    As speaker after speaker took the stage on Tuesday, I sank deeper and deeper into my seat, realizing that I had completely missed the mark—and as if that were not bad enough, several key analogies in my presentation had already been used by the first-day speakers. I did go to the party at Delirium Café, but left early to sleep the restless sleep of the nervous speaker. In the wee hours of the morning, I gave it up as a bad job, and spent several hours basically rewriting my entire talk in the light of what I had seen. Goodness knows what my neighbors thought of me rehearsing my lines, although I tried to keep the noise down.

    Here is the result in Slideshare, or you can watch the video here. That link should take you to the correct time stamp, but if not, scroll to about 5h30′ in.

    Katja Budnikov was also doing amazing sketchnotes (live!) of all the presentations. Here is the one from my session:

    Relief

     

    In the end it seemed to go down pretty well. I did my level best (honest!) to call on my own now-obsolete experience as a practitioner and on conversations with people who are more current than I am with the nuts and bolts of modern IT, rather than relying on the crutch of vendor boilerplate, and I think I was mostly successful in that.

    If you are working for a vendor and considering submitting to the Call for Papers at this sort of event, whether it’s Monitorama, DevOpsDays or Config Management Camp, I would definitely suggest test-driving your content as widely as possible. You probably have a lot of subconscious reflexes as a speaker that are not appropriate for the sort of audience these events attract. Even when you are conscious of them, it’s hard to eradicate them completely, but do your best.

    In the other direction, while I have enormous respect for the technical level of Monitorama attendees, some of them seemed to be at risk of losing track of the big picture. I would not want Monitorama to change its character as a techie’s conference, but I am grateful that we had a few presentations outside that mold, and I hope that I was able to provide some context for how all the wonderful technology gets used out there in the big bad world.

    — Dominic Wellington

  • Knowing What Your Microservices Are Doing

    Knowing What Your Microservices Are Doing

    Microservicin’ ain’t easy, but it’s necessary. Breaking your monolith down into microservices is a must in a cloud-native world, but it doesn’t automatically make everything easier. Some things actually become more difficult. An obvious area where it adds complexity is communications between services; visibility into service to service communications can be hard to achieve, but is critical to building an optimized and resilient architecture.

    The idea of monitoring has been around for a while, but observability has become increasingly important in a cloud-native landscape. Monitoring aims to give an idea of the overall health of a system, while observability aims to provide insights into the behavior of systems. Observability is about data exposure and easy access to information which is critical when you need a way to see when communications fail, do not occur as expected or occur when they shouldn’t. The way services interact with each other at runtime needs to be monitored, managed and controlled. This begins with observability and the ability to understand the behavior of your microservices architecture.

    A primary microservices challenge is trying to understand how individual pieces of the overall system are interacting. A single transaction can flow through many independently deployed microservices, or pods, and discovering where performance bottlenecks have occurred provides valuable information.

    It depends who you ask, but many considering or implementing a service mesh say that the No. 1 feature they are looking for is observability. There are many other features a mesh provides, but those are for another blog. Here, I’m going to cover the top observability features provided by a service mesh.

    Tracing

    An overwhelmingly important thing to know about your microservices architecture is specifically which microservices are involved in a user transaction. If many teams are deploying their dozens of microservices, all independently of one another, it’s difficult to understand the dependencies across your services. Service mesh provides uniformity, which means tracing is programming language-agnostic, addressing inconsistencies in a polyglot world where different teams, each with its own microservice, can be using different programming languages and frameworks.

    Distributed tracing is great for debugging and understanding your application’s behavior. The key to making sense of all the tracing data is being able to correlate spans from different microservices related to a single client request. To achieve this, all microservices in your application should propagate tracing headers. If you’re using a service mesh such as Aspen Mesh, which is built on Istio, the ingress and sidecar proxies automatically add the appropriate tracing headers and report the spans to a tracing collector backend. Istio provides distributed tracing out of the box, making it easy to integrate tracing into your system. Propagating tracing headers in an application can provide nice hierarchical traces that graph the relationship between your microservices. This makes it easy to understand what is happening when your services interact and if there are any problems.

    Metrics

    A service mesh can gather telemetry data from across the mesh and produce consistent metrics for every hop. Deploying your service traffic through the mesh means you automatically collect metrics that are fine-grained and provide high-level application information since they are reported for every service proxy. Telemetry is automatically collected from any service pod providing network and L7 protocol metrics. Service mesh metrics provide a consistent view by generating uniform metrics throughout. You don’t have to worry about reconciling different types of metrics emitted by various runtime agents or add arbitrary agents to gather metrics for legacy apps. It’s also no longer necessary to rely on the development process to properly instrument the application to generate metrics. The service mesh sees all the traffic, even into and out of legacy “black box” services, and generates metrics for all of it.

    Valuable metrics that a service mesh gathers and standardizes include:

    • Success rates.
    • Request volume.
    • Request duration.
    • Request size.
    • Request and error counts.
    • Latency.
    • HTTP Error codes.

    These metrics make it easier to understand what is going on across your architecture and how to optimize performance.

    Most failures in the microservices space occur during the interactions between services, so a view into those transactions helps teams better manage architectures to avoid failures. Observability provided by a service mesh makes it much easier to see what is happening when your services interact with each other, making it easier to build a more efficient, resilient and secure microservice architecture.

    — Zach Jory

  • AIOps: Managing the Second Law of IT Ops

    AIOps: Managing the Second Law of IT Ops

    Or, why IT systems keep breaking

    The tension between the two halves of DevOps is long-standing, going back long before the term was even invented. What underlies the split between Development and Operations is a fundamental difference, like an actual physical law. Let’s explore that analogy together.

    IT Ops is fundamentally a constant fight against entropy, meaning disorder. Sysadmins expend significant energy trying to keep everything up and running, in a process which can sometimes feel like trying to shore up a sandcastle while the tide is coming in. Systems are constantly breaking down; users need changes made that introduce risk; and, as if that wasn’t enough, the whole shebang is constantly under attack from malicious outsiders. It’s no wonder that sysadmins have a reputation for being a bit grumpy!

    Homer: In this house, we obey the laws of thermodynamics!
    It’s not just a good idea, it’s the law.

    In contrast, Dev actually increases the level of entropy. Every new release, be it as small as a one-line fix or finding that errant semicolon, is a change that has to be made, reducing the order that Ops fights so hard to maintain. Dev is entropogenic, embracing change as a positive force.

    The interaction between Dev and Ops is analogous to the second law of thermodynamics: Entropy can only increase over time, unless energy is introduced to the system from outside. In other words, unmanaged change will eventually cause something to break, unless people put in continuous effort to prevent those breaks.

    Thermodynamics: Lots of Heat, not Much Light

    These conflicting goals, taken to extremes, can get unhealthy, with Ops actively fighting change or Dev going around Ops to increase their velocity of change. Left unchecked, this tension can turn into a vicious cycle, as frustration with the “Department of No” drives developers into unmanaged “shadow IT.”

    The good news is that the analogy with hard laws of physics is not perfect. The process of increasing entropy is not irreversible, and does not require constant inputs of energy to prevent. New agile approaches to IT Ops are emerging that are light and flexible, taking their structure from that tension. In the physical world, this is called “tensegrity,” and was originally formulated by Buckminster Fuller, one of those annoying polymaths who seem to make breakthroughs in every field they touch.

    Tensegrity structure
    Tensegrity structure

    IT Ops needs its own equivalent to tensegrity, a flexible, holistic structure that is able to embrace change while maintaining its shape. Structures that try to resist change through rigidity are brittle and ultimately fragile.

    Today, this flexible strength is enabled by AIOps, a term coined by Gartner to describe a new way of operating IT, based on augmenting human skills with automated analysis enabled by modern algorithmic and machine-learning approaches.

    Complexity Breeds Complexity

    Modern IT is too complex for any single person or even team to keep track of, instead requiring multiple specializations. In turn, each of these functions requires its own specific support tools that enable that particular role. Beyond a certain point, it becomes very difficult to combine these partial views to gain a holistic understanding of what is actually happening in the environment.

    Because of the scale of modern IT environments and the sheer rate of change that businesses require, old-style rules-based “Managers of Managers” can no longer keep up, requiring more effort to build and maintain their worth. Especially as infrastructure becomes more and more self-modifying in response to changing conditions and demand, IT Operations needs to accelerate just to keep up.

    AIOps brings developing issues to the attention of sysadmins before they become visible to customers and end users, and gives them the tools to collaborate effectively between different teams and specialist roles to avoid outages or restore service quickly and efficiently. In this world, automation is used to augment the capabilities of the human operators to make them more effective, letting people concentrate on adding value, not just fighting fires.

    This is how Ops can get out of the business of fighting Dev, and instead become a supportive partner. AIOps can catch issues that are affecting services and flag them to the right people early on, including the DevOps engineers that are responsible for production support.

    For developers, one of the fears around DevOps is that in “picking up the pager,” they are opening themselves up to a constant stream of irrelevant alerts. Old Ops hands nod wryly at this, or maybe try to share their personal approach to filtering the noise. Instead, AIOps ensures that only valid, actionable alerts get as far as being shown to a human being and taking up their valuable time.

    Eyes on the Prize

    It’s not all roses. As the AIOps market begins to mature, there is a certain amount of “algorithm-washing” going on, with more and more vendors trying to claim that either their legacy tool was AIOps all along, or that their one-trick pony is actually a whole three-ring circus. What gets lost in the excitement is what all of those algorithms are actually supposed to achieve.

    Me too!
    Me too!

    In considering a shift toward AIOps, the goal should not be simply to build a better mouse trap, meaning a smarter filter for monitoring events. The success of the project can be measured not so much in the mathematical results of the algorithms, but in the smoother exchange between the human users. This easier collaboration is especially important across team boundaries, whether these be different specialist units within the Ops function or developers and software engineers who need to work with Ops to understand the performance of their code in production.

    The benefit of transforming operations this way—moving from sequential, waterfall Ops to abile, collaborative Ops—is to turn DevOps-enabled teams into a sort of perpetual-motion machine, able to continue operating without requiring constant massive inputs of energy from outside the system. Modern applications and the complex infrastructures that support them cannot be managed with the sorts of rigid, cobbled-together, industrial approaches that were appropriate in the mainframe days. AIOps is the agile, flexible approach that can not only accomodate that complexity, but thrive on it.

    — Dominic Wellington