devopsy.kinsta.cloud

Tag: monitoring

  • GDPR and Data Leakage: How Semantic Graphing Can Help

    GDPR and Data Leakage: How Semantic Graphing Can Help

    Many of the hallmarks of modern application development that drive efficiencies in the software development life cycle (agile, CI/CD, microservices, cloud infrastructure, open source libraries and third-party APIs) also create significant data leakage risks. Meanwhile, the EU’s General Data Protection Regulation (GDPR), which took effect May 25, creates a need to protect more types of data than ever before and comes with the strongest penalties for non-compliance: €20 million or up to 4 percent of a company’s global revenue.

    Preventing Data Leakage Getting More Difficult

    There are three primary reasons why data leakage has become more challenging for developers: complexity, speed and external services.

    Applications themselves have become more complex as organizations adopt microservices architectures. While this enables developers to be more efficient, it also often leads to more siloed views of the application while the data flows become more complex. For example, how many organizations understand  every entry point, exit point and handling of variables across every microservice in their broader application? And, organizations that adopt open source to increase efficiency by not having to reinvent non-differentiating code often fail to comprehensively understand the third-party code in depth. This makes understanding data flows even more difficult.

    The increasing pace of releases exacerbates complexity issues. Many organizations now release monthly, weekly or even daily. With less time to focus on anything beyond meeting feature deadlines, mapping data flows often becomes an afterthought. Furthermore, even if an organization is able to thoroughly map their data flows for any given point in time, the rate of change makes it impossible to stay on top.

    As organizations adopt third-party services for key functions such as code repositories, logging, databases and analytics, the consequences of leaking data become severe. For example, Uber leaked AWS credentials into GitHub, from which bad actors were able to initiate an attack that lead to 57 million records being compromised—simply by logging in.

    Why GDPR is Different

    GDPR is a significant expansion of previous data protection regulations in a number of ways. Unlike the Health Insurance Portability and Accountability Act (HIPAA) and Payment Card Industry Data Security Standard (PCI DSS) in the United States, GDPR applies globally and across industries. Any company that holds or processes data on an EU resident is subject to the regulation. Cloud-based software is inherently more global, and many U.S. or Asian-based organizations have users that are EU residents.

    GDPR expands the definition of what needs to be protected in a big way. HIPAA and PCI DSS formalized rules around protecting healthcare and credit card data. However, GDPR expands data that must be protected to include anything that can be used to identify a person—anything from a name, a photo, an email address, bank details, posts on social networking websites, medical information or a computer IP address.

    And GDPR has real teeth. Companies can be fined up to 4 percent of their annual global revenue or €20 million for major breaches and 2 percent of their global revenue for minor breaches. Given the global nature of modern cloud applications, the potential impact of GDPR is far-reaching.

    Traditional Approaches to Preventing Data Leakages

    Traditional approaches to preventing data leakage have been focused primarily on acting as a proxy between the application and the rest of the world. In such a position, the proxy can attempt to filter data and look for patterns. These approaches are outside-in. They do not attempt to understand the application itself; rather, they inspect traffic flowing from the application without context.

    First, let’s clarify that despite the similar sounding name, preventing data leakage is not the same as data loss prevention (DLP). Traditional DLP vendors such as Vontu (now Symantec) and cloud access security brokers (CASBs) are solving a different problem. They are an IT-centric solution to preventing malicious or accidental data breaches that are typically driven by end-users moving documents via email or file sharing. These tools aren’t designed to prevent the applications themselves leaking data and they are of little use to developers.

    The more application-specific attempts to identify data leakage have been via web application firewalls (WAFs). As proxies, WAFs often employ pattern matching techniques such as looking for 16-digit numbers on the assumption it may be a payment card, keyword matching, fingerprinting and other statistical content analysis techniques. In practice, these methods are ineffective because they create far too many false positives. Furthermore, encryption is problematic in that the WAF either can’t inspect the data or a man in the middle decryption must be set up, which is computationally expensive and can add latency. Latency may not matter for traditional DLP solutions monitoring email, but it’s vitally important to developers managing cloud applications.

    One traditional method for identifying data leakage that does take and inside-out approach is taint tracking, which evaluates pre-existing binaries. In reality, taint tracking is impractical because of excessive performance costs and numerous false positives due to taint proliferation (transitively from application to web framework to kernel).

    Taint and information-flow analysis approaches track the flow of data through the program, assuming that the data originated from untrusted sources. Sources are where such data flows enter the program, and sinks are where they leave the program.

    Several publications define sinks informally as “data that leaves the system,” which is too imprecise to train a machine-learning based classifier; such classifiers are only as good as their training data.

    Dynamic taint tracking by plain instrumented emulation is extremely expensive and tend to slow systems. Such a slowdown is unacceptable in practice and significantly hinders the adoption of dynamic taint tracking systems for everyday use. Such taint proliferation can substantially impair the performance of the running system.

    Mapping Data Flows with Semantic Graphing

    To truly prevent data leakage, you must first map data flows. You must understand the building blocks of the application, including its security DNA. However, the difficulty comes from balancing comprehensiveness with speed. Simply put, to account for all possible leakages in a rapidly evolving application you must automate.

    Semantic graphing is based on a simple observation: There are many different graph representations of code, and patterns in code often can be expressed as patterns in these graphs. It is a graph of graphs that, for the first time, enables automatically creating profiles that understand the context of an application. Semantic graphing is language-neutral intermediary representation designed for code querying, which focuses on what the program is designed to do, what it is not supposed to do and how it functions. Because semantic graphing identifies which variables are sensitive and how they flow across each microservice, it then becomes possible to visually represent data paths and identity leakage/violations.

    In this instance, we seek to understand data flows and prevent leakages. However, the ability to understand an application’s context in code is fundamentally the foundation for DevSecOps. AppSec remains manual primarily because humans must apply the application’s context either through configuring rulesets or wading through false positives, both of which are impossible tasks. Beyond just data leakage, semantic graphing has the potential to deliver security at the speed of DevOps.

    — Chetan Conikee

  • DevOps Chat: Modern Applications with SignalFx CEO Karthik Rau

    DevOps Chat: Modern Applications with SignalFx CEO Karthik Rau

    Monitoring has come a long way since the days of 2 a.m. alerts, with automation and analytics now integrated in—and integral to—the life cycle of modern applications. It’s a topic Karthik Rau, CEO of SignalFx, knows well, and one we discuss in this latest DevOps Chat.

    In its three years of selling, SignalFx has grown a whopping 300 percent year over year—an astounding number, even for a startup.

    As usual, the streaming audio is immediately below, followed by the transcript of our conversation.

    Alan Shimel: Hi, everyone, it’s Alan Shimel and you’re listening to another DevOps Chat. Today’s DevOps Chat features Karthik Rau, CEO of SignalFx. Karthik, welcome to DevOps Chat.

    Karthik Rau: Thanks, Alan. It’s great to talk to you again.

    Shimel: Always a pleasure to have you on. So, Karthik, let’s just do a little setting the base here, baseline setting. And I’m gonna assume most of our listeners have at least heard of SignalFx, but maybe some haven’t and maybe some have heard it but they’re not sure exactly of who SignalFx is. Why don’t we start there?

    Rau: Sure, Alan. SignalFx is a monitoring and operational intelligence solution for the developers and operators of cloud-native applications. Our founding team, my cofounder and our original engineering team, all came out of Facebook and were responsible for building the monitoring systems at Facebook, back when they had a “move fast and break things” culture, which today everyone calls DevOps, but, back then, they didn’t really have a term for it. So, at a high level, what we do different from everyone else is we are focused very heavily on the analytics part of the monitoring problem. We collect data – time series, telemetry data – across the customer’s entire environment, from the hardware to the middleware to the application, all the way up even through the API layer, and we apply real-time statistical models around that data to identify patterns that are unusual and different, and alert on those trends proactively so that customers can get ahead of issues and be more proactive in their monitoring, rather than reacting to a bunch of noisy alerts.

    So our customers are able to, No. 1, get a much better handle of their distributed cloud-native application architectures, understanding the patterns, what’s happening. They’re able to detect issues as they emerge, which is very important in today’s environments because most people are pushing changes out a lot more aggressively, and so you wanna detect changes immediately, rather than wait and discover stuff after you’ve made a number of changes. And it’s particularly important for today’s application architectures where you’ve got a lot of bursty workloads, like with containers, where you’re spinning up and spinning down capacity in minutes and you really need your monitoring systems to be (a) real-time and discover components immediately and then (b) be a lot more sophisticated in finding issues as they happen.

    Shimel: Excellent. And, Karthik, you know, I should also mention that, in terms of UI, SignalFx is one of the best UIs, web-based UIs that I’ve ever seen. And I’ve looked at a lot of applications, a lot of programs, over the year – over the years. Year? I wish it was “year.” And, you know, so – and I’ve told you this before. I’m just think whoever designs the interfaces there is a rockstar, so just mention that, but –

    Rau: Yeah. Thank you.

    Shimel: You know, we spoke a little bit off-mic, Karthik, and we were going over what are some of the new metrics coming out of SignalFx, how business is going, and you mentioned some really eye-popping numbers, right? First of all, you guys are showing something like 300 percent year-over-year growth for the last couple years, which is – you know, everyone will say, “It’s a startup; they’re supposed to grow like that,” but, dude, until you’ve done it, you haven’t done it, so that’s an astounding number. But also a big move into large enterprises. Is that correct?

    Rau: Yeah. So, yes, absolutely. So we’ve been, as you’ve mentioned, tripling our business year over year, since we – over the past year, but significant growth even above that in the early years. We’ve only been selling for about three years. Our customer base is focused quite a bit on mid- to larger-size companies, so we’ve always been focused on larger accounts, whether that’s the web-scale players, who are operating large-scale applications with four or five nines availability requirements, but also a lot of the enterprise accounts, who what we’ve been really excited about is they’re really moved wholesale into cloud and cloud-native application architectures. And we’ve seen that trend accelerate over the last two or three years, so that’s been a large area of success for us. You know, we’ve got Fortune 500 customers across many verticals – media, retail, financial services, industrials, transportation – so we’re definitely seeing that trend accelerate across the enterprise, not just in a handful of verticals.

    Shimel: Absolutely. But, Karthik, what about for the person out here listening to this, right? And he says, “Wow, that sounds great. I love what Karthik says they’re doing and the Facebook kinda background and all of this great stuff, but I’m not a unicorn. I’m not even a large enterprise.”

    Rau: Well, I think that everyone today is trying to build software quickly, right? Gone are the days where you have a three-year development cycle and you release software and hope you get it right. It’s just not gonna work in today’s world of web services and your competition are tech companies that are moving at lightning speed. So the issue isn’t really – the world is changing. The way people are building and releasing and managing software is fundamentally different.

    And what SignalFx does, the analytics around SignalFx, you can look at it in two ways. One way is, yes, if you’re a large environment, you absolutely have to have it to reduce noise, but another way of looking at it is you could be a small environment and you wanna have a more predictive way of discovering issues, as they emerge, so that you too can move fast without breaking things, right?

    And if you look at it that way, having the kinds of capabilities that SignalFx provides are critical, even if you’re a department within a big company or you’re a startup with 50 people or 20 people. You still wanna be able to do the proactive analysis of, “Hey, this code push that I just rolled out, a canary release, is unusual relative to the rest of my application. And maybe that’s indicative of a problem with my code and I wanna roll it back before rolling it out across all of my users.” Right? You don’t have to be Facebook-sized or Google-sized to have that problem. You’re just building a modern application.

    So that’s why we do have customers across segments. I mean, our focus is certainly on the larger accounts, but we’ve got a lot of departmental deals within enterprises. We’ve got a lot of startups that are also using us, who care about having a more predictive approach to monitoring.

    Shimel: Yeah, no, you know, I think the key there – and you mentioned the word, Karthik, and it’s funny; it’s a word I’ve been hearing over and over again. And that this term of the “modern application.” I don’t care whether you’re big, small or Goldilocks. The modern application, as we call it, right, demands certain things, and one of them is obviously monitoring, but also modern applications demand automation, real-time type of data flows and insight. And it’s not just the guys who develop once every few years – that certainly is not a modern application – but it’s also more than just how frequently you update your code. I think the modern application, you know, what goes are in, the kind of stuff, or at least one element of it is the kind of stuff you’re talking about with SignalFx. Right? These go like hand in glove.

    Rau: Yeah. Absolutely.

    Shimel: So I think that’s – so, to me, more than size of company. I mean, that’s almost a determining factor. Another item I just wanted to mention on the – or bring up on SignalFx itself, as a company, is you guys recently announced a rather large round of funding as well.

    Rau: That’s right, Alan. On the backs of all of our success, we continue to build out our product and continue to accelerate our go to market. We just closed a $45-million Series D round in May and we’ll be using that to continue our investment in R&D. We’re continuing to expand globally. I think I’d mentioned to you offline here that we’ve just expanded into Europe. We also recently expanded into Australia. We’ll continue our expansion geographically, so, hopefully, a lot more people across the seas will hear about us and see us, in the coming quarters.

    Shimel: No doubt in my mind about it, Karthik. I can tell you, from – when I look at devopsy.kinsta.cloud’s demographics and metrics and analytics, it blows me away that only about 36, 35 percent of our audience these days are U.S.-based.

    Rau: Mm-hmm.

    Shimel: About 31, 32 are based in Europe. 15 percent in India. Australia is probably the fourth or fifth largest country, for instance, for our readers. So –

    Rau: Yeah, that’s right.

    Shimel: – you know, the thing about this market is it’s truly, truly a worldwide market. And I know, Karthik, if you’ve had the chance to go to China recently, but I was there for Interop China last month, talking about DevSecOps, and I will tell you that the DevOps market in China is exploding. I mean, just there’s a lot of people there, obviously, as there are in India, but DevOps is really catching fire.

    And, as you and I were talking about offline, Karthik, it’s one thing if you’re chasing that crowd or you’re chasing that wave. In the case of SignalFx, it’s a beautiful thing, where, maybe when you did launch, you were a little out ahead of the market, a little out ahead of the state of art, but they’re catching up to you now, right? They’re coming. And it’s good to be there, waiting for them.

    Rau: Yeah, absolutely. The market – what’s been most interesting about – you know, at SignalFx the last several years, has been, when we launched the company three years ago, our – what we did that was so different from everyone else is we had this real-time streaming analysis engine, streaming analytics engine, that could apply these sophisticated models within seconds across large data sets and help you be a lot more predictive in identifying issues. And no one else was doing it that way and still no one else is doing it that way. And, at the time, people would ask us questions of like, “Well, what does it really matter if you can do things in seconds versus minutes? ‘Cause the human attention span is only minutes and if you wake up in the middle of the night, it might take you minutes to go and deal with an alert.”

    Well, that was kind of in the old days. Now what’s happening is there’s so much more automation happening across the entire operations landscape. I mean, people talk about it as infrastructure as code, right? You basically automate the entire end-to-end process as much as you can. And, more and more, there are these frameworks that enable greater automation, like Kubernetes, and what we’re finding now is people are really recognizing that, “Maybe we don’t need roomfuls of NAC engineers waiting to find a yellow light or a red light to then go and manually invoke a script from a run book, if we can just automate it.”

    Well, guess what? If you’re gonna automate a remediation like that, garbage in, garbage out. You need quality in. You need to have quality signals. You need to be able to untease signal from noise and you need to be able to do it quickly. And if you can do it quickly, then you can automate quickly and you don’t – the whole notion of having a human wake up and spend minutes orienting themselves goes out the door ’cause you can just – or having the roomful of NAC people responding to it goes out the door. You can just automate it. And then the capabilities of SignalFx then are very unique and very critical to enabling you to drive this end-to-end automation.

    And what we’re finding is now more and more of our customers and marketplaces recognizing this, as they’re embarking down this path of leveraging microservices and automation around microservices and infrastructure as code. Having a system like SignalFx, with its real-time analytics, is absolutely critical. So it’s been fascinating to see that evolve over the last few years.

    Shimel: Yeah. No, for sure. You know, Karthik, it was interesting. I saw an interesting metric today. Something – I forgot who it came out of – Gartner or someone. Within two years, something like two-thirds of cloud deployments will be container-based, which kinda blew me away, to tell you the truth. Have you heard anything like that?

    Rau: I mean, that’s an incredible metric. I mean, I would say we’re certainly seeing a tremendous amount of interest in containers. Well, it’s gone from interest to actual deployments, right? So, two years ago, there was a tremendous amount of interest in it. Nowadays, I would say that is the de facto new architecture, so I think the keyword there is “new,” right? So two-thirds of new deployments, I totally believe that, that that will be the case, 100 percent.

    Shimel: Crazy. I mean, we talk about internet time and the time crunch and it seems to be accelerating like global warming is and climate change is, but the idea that we’d see that kind of number in that kind of time just kinda blows me away. So, Karthik, we spent a bunch of time with this stuff, going over the SignalFx, a little bit about the market, right? Let’s take a look at a crystal ball and gaze into where do you see SignalFx, let’s say, going over the next, let’s say, 18 tp 24 months?

    Rau: Well, we continue to invest across our business, right? So our focus right now is we believe there’s a giant problem that needs to be solved in the industry, which is, “How do you get better visibility and monitoring around the new stack? And then, beyond that, how do you get better visibility into how your operations are impacting your business?” So, as I mentioned at the top of this podcast, we think of ourselves as monitoring and operational intelligence for modern applications, so one of the things that we – we see all our customers go through a similar progression, where you’ll start doing the infrastructure monitoring for this new world, where you gotta look at the entire microservice, versus an individual node, and you start to get into application and look at application performance issues.

    But then, very quickly, customers typically start to look at how are all of these metrics affecting their business, right? You know, “How many of my partners are making API calls in my services? And who are my most expensive partners and my least expensive partners?” Or, if you’re a SaaS vendor and you wanted to do a gross-margin analysis by customer, all of the data already exists in your monitoring system, and we see more and more of our customers extending the use of these systems to answer broader questions around operational intelligence.

    So that’s kind of – those are the three sort of key pillars of what you use a system like a SignalFx for, right? It’s infrastructure monitoring, it’s application monitoring, and then it’s operational intelligence. So our plans, as a company, are really focused around those three pillars. You need to invest in product to support all of the use cases around those three areas. And then continuing to invest in our sales and marketing team so that more and more companies out there can benefit from what we have to offer. So that’s our focus over the next 18 to 24 months.

    Shimel: Fantastic. And, as we spoke about earlier, it’s nice to be out in front of the market, but, Karthik, your job, as the CEO, if you choose to accept this mission impossible, is how do you stay out front of it? You’re right? How do you know – how do you – I mean, you think containers are gonna be dominant, right, so you wanna put that cloud-native dominant. What do you think is sort of next big things coming out there?

    Rau: Well, in our space, I think the most fascinating new set of technology is around service mesh. You know, as more and more of the world moves toward leveraging containers and orchestration frameworks like Kubernetes, the service mesh is gonna be a really interesting way of automatically getting visibility into all of the flows between all of these different microservices, having a control plane to operate against all these microservices. And it’s still relatively young. It’s still in its infancy, but there’s a lot of great stuff happening there, you know, what you see Lyft doing with Envoy and Istio and LinkerD. There’s just a lot of really interesting stuff happening there.

    And the theme here is, as developers own more and more of running their applications and the runtime, what you’ll find is more and more of the stuff that used to be this black-box, proprietary thing that vendors would come in and do in the runtime are just becoming available as a part of the core services. So, in the case of service mesh, a lot of the instrumentation of metrics and traces and monitoring data is just you’re gonna be able to get it by virtue of running in a Kubernetes-style environment and you’re not gonna need these expensive, proprietary agents to come in and collect that data for you.

    So that’s great for customers. It creates a standardized mechanism of data collection and it provides a lot of automatic instrumentation, which is great for customers. Of course, then that just begs the next question, “Well, what are you gonna do with all that data?” So we’re focused on ingesting more and more of these new types of data from these new frameworks and making it even more – the more data we get, the more powerful and rich the context we can provide around our alerts and algorithms and dashboards, so we’re really excited to see that trend continue.

    Shimel: Very cool. Karthik, we’re about out of time. We’re probably over time here. I apologize for keeping you on so long. But, hey, man, you guys are kicking butt and taking names, and congratulations. For anyone listening out there who hears Karthik talk about growing 300 percent at SignalFx year over year and raising $45 million and it sounds like, “Well, la dee dah,” I’ve been there and I can tell you it’s not easy. So congratulations. Keep up the great work and we’d love to check back in –

    Rau: Thank you, Alan.

    Shimel: Let’s not wait a year to check back in with SignalFx. Let’s have you on here sooner than later.

    Rau: Absolutely. Let’s do it again soon, Alan. Sounds good.

    Shimel: Karthik Rau, CEO, cofounder of SignalFx, our guest here on DevOps Chat. You’ve just listened to another DevOps Chat. Have a great day, everyone.

    — Alan Shimel

  • An Easy Way to Optimize Network Performance Monitoring

    An Easy Way to Optimize Network Performance Monitoring

    Network performance monitoring, and especially network optimization, is more of an art than a science because there are so many factors that figure into network and application responsiveness. In addition, because there is a vast amount of data traversing the network, determining the right kind of data you need to monitor (and where you should you be capturing the data from) can become very difficult. As an example, according to research conducted by Enterprise Management Associates in October of 2016, 41 percent of IT personnel spend more than 50 percent of their time working on network and application performance problems.

    The data collection process is further complicated by the fact that tactical data loses up to 70 percent of its value after 30 minutes. This makes the speed and accuracy of data analysis critical if you hope to resolve your network or application issues.

    There is one easy way to improve your lot in life, though: Add a network visibility architecture. Most IT personnel have a security architecture and a network architecture, but very few have a visibility architecture. Network visibility is what enables you to capture the right data at the right time, so you can do something useful such as quickly isolate potential problems.

    Setting up a visibility architecture is straightforward. There are four main components:

    • Data access
    • Data manipulation
    • Application intelligence
    • Purpose-built analysis tools

    Data access can be provided by physical taps, virtual taps and bypass switches. These devices give you timely access to the data you need. A SPAN port also can be used; however, there can be issues with the quality of the data from this type of device, so it is not recommended.

    After you have acquired the monitoring data you need— probably along with a bunch that you don’t need—the next step is to send it to a network packet broker (NPB). This device provides granular filtering capability to maximize the flow of relevant information to your monitoring tools. Specific types of data can be segmented out and sent to specific tools to improve efficiency and faster time to resolution. NPBs can provide the following services as well: data aggregation, deduplication and load balancing of Layer 2 through 4 (of the OSI model) packet data.

    Application intelligence is another capability available through an NPB. This functionality provides additional filtering and analysis at the application layer, i.e. Layer 7 of the OSI stack. For instance, you can leverage application intelligence to prevent application bandwidth overloads on your network. You can literally see the amount of traffic on your network based upon application type (e.g. Hulu, Netflix, Pandora, FTP, HTTP, RTP, etc.). You also can use it to identify slow or underperforming applications.

    The final layer of the visibility architecture is made up of your security and monitoring tools. These devices typically are special-purpose tools (e.g., sniffer, network performance monitoring [NPM], application performance monitoring [APM], etc.) that are designed to analyze specific data. For instance, APM tools can provide real-time data analytics to help you manage your network and lets you see problems before your users do.

    However, standalone deployments of these tools can run into different problems, such as overloaded disk space and processing, the need for different interface ports based upon network traffic speed and the need for lots of input ports to capture data across the network. An NPB can be used to capture network data and filter that data before it goes to the NPM tool. This increases the efficiency of the tool by reducing clutter. The additional filtering of duplicate data further enhances the efficiency and also removes the storage waste associated with storing irrelevant data.

    Network administrators need network visibility solutions to help them discover, isolate and solve problems related to the network and its applications. This architecture gives your network tools a complete picture of the network so they can resolve issues faster.

    — Keith Bromley

  • What to Consider in a Full-Stack Monitoring Solution

    What to Consider in a Full-Stack Monitoring Solution

    In the DevOps space, there are various tools available for monitoring. When building a full-stack monitoring solution, however, selecting the right monitoring tools and designing monitoring based on IT operations’ needs is a critical decision.

    Design Considerations for Implementing a Full-Stack Monitoring Solution

    The following are some of the key design considerations while implementing full-stack monitoring:

    Distributed Monitoring

    Consider choosing the topology that is best-suited to the monitoring needs of the applications. For example, small teams with fewer applications to monitor can choose to implement monitoring tools in a single-server standalone topology, while large IT operations teams with many different monitoring applications can choose a distributed architecture in which the master nodes contain metadata of the monitoring tool, while monitoring and slave nodes are centralized and act as the data collectors and data nodes, then connected together as a centralized monitoring system. Monitoring tools clusters should contain a minimum recommended master nodes and data nodes for high availability. If a network outage occurs, monitoring systems may not be able to detect the underlying problems; however, with distributed slaves, if one slave is offline, a new slave node can be added to the cluster, which performs checks and writes the results back to the data nodes. Choose on-premises or SAAS-based monitoring tools depending on your workloads (cloud/on-premises). Pricing also varies and should be considered.

    Availability

    Site reliability engineering (SRE) teams focus on the reliability, uptime and availability of the applications and thus monitor continuously to identify any potential risks/outages. This mandates that the monitoring tools to be up and running all the time. Most monitoring systems are set up in the same network they monitor, and if a network outage occurs, the monitoring system can’t monitor for security threats. Thus, monitoring tools must be configured inside a high-availability cluster, with availability zones and geo-redundancy regions for availability, backup and disaster recovery.

    Scalability

    It is common to have multiple monitoring systems to monitor multiple locations or data centers with thousands of hosts. Data from each host is collected multiple times daily, which can result in huge load and performance hits. In a distributed monitoring system with slaves distributed, data checks can be performed locally in the data center, the data can be processed and slaves can then send the results to the master. Based on the load, the hardware CPU cores can be utilized to run multiple checks in different threads or slaves can be added to the cluster for scalability and redundancy.

    Storage

    Monitoring systems collect and use data from several workloads for various monitoring needs. This results in continuous monitoring, which generates volumes of data accumulation and can result in slow performance of the monitoring tools to analyze the data, demanding high storage needs to store the data and impacting faster troubleshooting. Monitoring systems use databases that are document-based or time series-based. Time series-based databases can store data with time as points, and modern monitoring systems offer plugins to collect data from various sources and provide a built-in schema of the metrics to be stored to the database. Consider customizing the plugins depending on the metrics needed to be stored, as well as aggregation of data based on the granularity and the necessary data storage timeframe. Data can be aggregated and stored in weekly or monthly tables in the database to minimize the overhead of maintenance cost of the monitoring tools. Configure retention and backup policies to retain the aggregate data for machine learning and anomaly pattern detections.

    Performance

    Performance of the monitoring tools should be tested, given the size and capacity of the master and slave monitoring nodes and the number of hosts to be monitored. Performing various checks in the hosts and then returning data to the server nodes involves lot of processing with high compute needs. Use queues or a transport mechanism to manage the data transfer from the clients to monitoring systems to avoid data loss due to network outage or latency. Most monitoring systems offer a built-in or support a queuing mechanism that performs retries to read/write operations in case of data loss. The slave agent nodes can improve the performance of the monitoring systems by performing data enrichment and data transformation locally, then forwarding the results to the master agents for better and faster reporting. Decide the time intervals you need to monitor the data based on the SLOs defined to optimize the performance of monitoring tool. Monitoring systems can use agent-based or agentless monitoring approaches: In an agent-based system, the data is collected using binaries installed in each host to be monitored, while an agentless monitoring system uses a pull-based approach to collect data from individual hosts. When using an agent-based approach, consider the memory footprint of the agent installation: Use optimized queries for search capabilities/indexes on the primary fields and aggregations in the monitoring system database to improve performance of the data retrieval, graphing and reporting.

    Security

    Configure security with permissions in the monitoring systems. The slave nodes/master systems should communicate securely with each other and access the endpoints and various ports of different processes for performing checks. Configure security at all levels of applications, different components, middleware software and firewalls to allow access to the ports and processes the monitoring systems need to access and monitor. Limit the security access to the defined stakeholders as per the need and administrator permissions to configure the monitoring tools for policies, thresholds, integration with other tools, managing users, backup, retention and other factors.

    Automation

    Because monitoring systems use either agent-based or agentless monitoring modes to collect data, agent binaries or plugins must be installed in every host to be monitored, which can be a tedious and time-consuming task. While a few of the monitoring tools can perform auto-discovery of the hosts and the agent’s binaries, custom configurations can be pushed to the hosts through configuration management tools and configured automatically to perform the checks and forward the instrumentation to the centralized monitoring system. The infrastructure scripts for the monitoring systems need to be version-controlled, and scheduled jobs and maintenance activities—such as cleanup of old data and running enabling batch scripts on the monitoring data to trigger remediate actions—can be deployed using configuration management and CI tools, reducing much of manual effort and maintenance costs. Configuring the monitoring of provisioning, de-provisioning and use of resources can be done through automation scripts in the monitoring tools.

    Testing the Monitoring tools

    Test your monitoring tools as you would any other application. Create test cases to test the configurations used for various checks; test whether right data has been reported from the right host with appropriate labels and tags; check the endpoints and the policies configured; test the latency/data transfer between the agents and master nodes; test whether the data correlation provided, if any, are giving the accurate information with time-bound; test to ensure the dashboards, alerts and visualization graphs are reporting proper data; as well as any machine learning and remediation scripts functional testing. Configure test environments to install monitoring tools in the staging environments to test the behavior, performance and the accuracy of the monitoring data before moving to production environments.

    Choosing Metrics to Set Up

    Metrics provides insights into the system behavior and choosing metrics that are not relevant can add more overhead and poor monitoring. Choose the metrics that provides the health of the overall platform such as throughput, latency and error metrics that can give insights of erroneous results from several sources, success metrics and performance metrics. Configure metrics for individual resources that measure utilization, saturation, errors, availability, performance such as response times, disk/CPU/memory usage, packets inflow/outflow, etc. Choose metrics to configure based on the events logs, such as scaling up/down hosts, containers up/stopped, build or deployment failures, profiling and stack traces, which can provide data correlation, and configure the metrics to enable drill-down from the work-level metrics to resource-level metrics to the event-level metrics, with required granularity and tagged to the scope of the resource. For an efficient full-stack monitoring, configure all the types of metrics to enable faster troubleshooting.

    Defining SLOs

    SRE teams focus more on the uptime, reliability and availability of an application and many factors can impact the reliability. Most commonly, application design can impact the performance of the application. The service-level objectives (SLOs) have to be designed with the desired objective and setting up target for the objective, with a clear definition of how the objective is measured and the conditions under which they are valid. Configure SLOs related to performance and availability that can be measured within the SLA with defined error budgets, and standardize the service-level indicators that can be created as templates for reuse across multiple projects.

    Designing Alerts

    Monitoring systems should know what to monitor, how to alert and what channels to use to communicate the alerts. Alerts can be defined on various criteria such as monitoring the potential issues/events and alert them before they occur (host, disk, memory, CPU metrics); monitoring the expected system behavior and key user actions for green status and alerting if the system behavior deviates monitoring the critical resources and components of the application and its impacts that affect various other parts of the applications and alert if the resources fail. Alerting also could be based on black-box monitoring, in which the alerts are configured based on externally visible factors and real users’ impact, or based on white-box monitoring. Check historic values and remediation actions in the past to determine the threshold values and the priority levels, where high-priority alerts can be sent as pages to on-call engineers and low/medium priority alerts are sent as secondary notifications. The alerts should be designed to provide contextual information such as links to appropriate graphs/logs/attachments that can contain deep information about the events occurred along with correlations. Identify the right team to receive the alerts and the communication channel that best suits the engineers in on-call rotations and responders with escalation plans and mitigation plans. Configure the frequency and the priority level of the alert based on the use cases. Striking balance between low-alerting and overalerting is important and alerting needs to be continuously improved and fine-tuned to avoid false-positive alerts.

    Visualization and Dashboards

    Metrics are collected and stored as time-series data, wherein data can be best represented as a graph with time on the x-axis and one can notice the value changes over time. Configure graphs and dashboards to provide a holistic view of different components that can be visualized over various time scales to understand long time trends as well as recent changes that might be affecting the systems currently. Configure generalized dashboards to visualize the current health metrics as on-screen displays and focused dashboards to drill down with contextual information that can be used for troubleshooting and specific use cases such as capacity planning.

    Incident Remediation

    Designing a well-defined incident remediation process involves integrating monitoring system alerts to create an auto-ticket in the IT service management (ITSM) tools and posting which incident/problem management workflow must be triggered. Identify the potential use cases for auto-remediation, in which a reliable source can identify the potential problem and get the same responses without any manual input for decision-making. For example, a workflow can be designed to integrate with knowledge base to check for known issues and from known components or sources to validate the issue and run auto-remediation actions and notifications bound with SLAs. Extending monitoring systems for auto-ticketing, auto-investigation, auto-remediation and auto-notification through multiple communication channels to the on-call engineers will accelerate full-stack monitoring.

    Machine Learning Analytics

    With volumes of data being generated and stored in monitoring systems, consider machine learning techniques such as predictive machine learning, in which algorithms learn how system behaves normally, provides a resulting model of normal behavior and predicts a range of observation points that lie outside the range that are marked as anomaly in the data, which can be used for capacity planning use cases. Design test cases for testing the machine learning algorithms to avoid any false-positives that incorrectly identify anomalies or result in malicious events going unnoticed as normal behavior.

    Understanding the needs of monitoring in your environment and then choosing the right tools, and implementing them with various design considerations in place, with continuous monitoring and optimizing further, can accelerate SRE (IT operations) teams to achieve full-stack monitoring.

    — Lavanya Subbarayalu

  • Benefits and Drivers of Full-Stack Monitoring

    Benefits and Drivers of Full-Stack Monitoring

    As organizations undergo digital transformations, there is increasing need to modernize the way IT operations is managed and to deliver applications faster, while at the same time delivering business value by managing applications’ availability, performance and outages in a cost-effective way.

    Because outages have a huge impact on revenue, there is a growing demand for site reliability people and monitoring systems, which can enable them to handle applications’ uptime, performance and reliability with a proactive and predictive monitoring approach.

    This article series focuses on the need for full-stack monitoring in today’s digital operations, the considerations for selection of monitoring tools and design considerations for implementing a full-stack monitoring solution.

    Why is Full-Stack Monitoring Necessary?

    Full-stack monitoring refers to the capability of monitoring all of the following:

    1. Infrastructure monitoring—Monitor servers, network, containers, cluster orchestrators, cloud, middleware, app servers, web servers and databases.
    2. Application performance monitoring—Perform log monitoring and analytics, synthetic transactions, sql transactions, data correlation with metrics and logs, end user experience monitoring, profiling and transaction-level tracing with code-level visibility.
    3. Performing end-end monitoring across the workloads in on-premises/cloud/hybrid environments through a centralized dashboard/visibility across the stack.
    4. Provide alerts, analyze, act and remediate the incidents faster, using the instrumentation data leveraging IT service management (ITSM), ChatOps and collaboration tools along with monitoring tools.
    5. Perform Analytics on the data being used to derive insights and predict demands.

    The following are some of the needs driving enterprises to implement full-stack monitoring:

    • Business agility: Business needs to deploy very frequently to sustain the market demands
    • Application reliability: Site/application platform is expected to be reliable, available and scalable, as outages impact revenue and may result in losing business and customers.
    • Bandwidth bottlenecks can impact the application performance, leading to potential business loss and needs network-level deep monitoring.
    • Failure can happen at any level and go unnoticed, leading to overhead in recovery and remediation; hence, monitoring from various sources and at different angles with varied levels are needed.
    • Application complexity and dependencies: Applications use multiple-third party components, multiple hosts and several underlying services, components and infrastructure. If any of these do not function as expected, it can degrade an application’s performance and result in issues/defects going unnoticed—and often reported by end users.
    • Cloud born applications, which are multi-tiered and multi-layered, use mix workloads (PaaS + IaaS) and monitoring of different sources/workloads has to be done for distributed hybrid environments (multi-cloud).
    • Use of virtualization and container technologies, which have enabled environments with dynamic and ephemeral states and such workloads, need monitoring at the orchestration and container levels.
    • Lack of end-to end monitoring tools and lack of visibility across all IT assets and workloads, which has increased the need of centralized monitoring data. The SRE teams need to get stakeholders’ buy-in based on the monitoring data to push back on the bad releases and take calculated risks. They also need to cater different types of monitoring data to different stakeholders and line of business (LOB) owners and overcome cultural challenges to get stakeholders’ buy-in.
    • IT operations needs to forecast an application’s behavior, capacity and performance, based on how the system performs in the given conditions. Machine learning of monitoring data and patterns can help predict capacity and anomaly.
    • Large volumes of data are collected from multiple sources giving out metrics, logs, etc., for various components of the stack. The IT operations team needs in-depth information about the components and dependencies along with metrics to provide contextual/correlation information. They need to correlate the data to determine symptoms and causes for faster troubleshooting. As such, they need monitoring tools that can provide correlation out-of-box to reduce the effort involved in triaging.
    • Configuration changes: Deployment of application/infrastructure/services are done through infrastructure as code automation scripts, which may contain issues in configuration. Lack of testing infrastructure, services and application end to end can increase the number of issues to be resolved; hence, there is a need of monitoring right from provisioning of resources, usage, maintenance and de-provisioning of resources.
    • Lack of end user perspective and business metrics from traditional tools, coupled with the need to eliminate toil and perform faster triaging/post-mortem and automated incident resolution, demands use of new-age full-stack monitoring tools.

    Growing business demands and agility, application reliability, deriving business insights and end user experience analytics are the key drivers for implementing full-stack monitoring. Hence, the IT operations traditional approaches are rapidly being transformed to digital operations (SRE operations), along with DevOps practices being widely adopted across enterprises.

    In the second and third articles in this three-part series, we’ll look at the considerations for selection of monitoring tools and design considerations for implementing a full-stack monitoring solution.

    — Lavanya Subbarayalu

  • Monitoring Your Data Center Like a Google SRE

    Monitoring Your Data Center Like a Google SRE

    Back in the early ’00s, when Google was beginning to expand its portfolio of services beyond search, it encountered a combination of challenges. Some of these emerged from familiar, classic disconnects between developers and operations folks or between IT services and line-of-business owners. Others were brand new, never-before-seen failure modes that arose from providing services on novel cloud platforms—and doing so at planetary scales. 

    To confront these challenges, Google began evolving a discipline called Site Reliability Engineering, about which the company published a very useful and fascinating book in 2016. SRE and DevOps (at least the contemporary version of DevOps that’s expanded into a vision for how IT operations should work in the era of cloud) share a lot of conceptual and an increasing amount of practical DNA; particularly true since cloud software and tooling have now evolved to enable ambitious folks to begin emulating parts of Google’s infrastructure using open source software such as Kubernetes. Google has used the statement, “Class SRE implements DevOps” to title a new (and growing) video playlist by Liz Fong-Jones and Seth Vargo of Google Cloud Platform, showing how and where these disciplines connect and nudging DevOps to consider some key SRE insights.   

    Understanding Google SRE Principles

    First, some basic principles: 

    Failure is normal – Achieving 100 percent uptime for a service is impossible, expensive or pointless (e.g., given the existence of masking error rates among your service’s dependencies). 

    Agree on SLIs and SLOs across your organization – Since failure is normal, you need to agree across your entire organization what availability means, what specific metrics are relevant in determining availability (called SLIs, or service level indicators) and what acceptable availability looks like, numerically, in terms of these metrics (called the SLO, or service level objective). 

    Use agreed-upon SLOs to calculate an “error budget” – SLO is used to define what SREs call the “error budget,” which is a numeric line in the sand (e.g., minutes of service downtime acceptable per month). The error budget is used to encourage collective ownership of service availability and blamelessly resolve disputes about balancing risk and stability. For example, if programmers are releasing risky new features too frequently and compromising availability, this will deplete the error budget. SREs can point to the at-risk error budget, and argue for halting releases and refocusing coders on efforts to improve system resilience. 

    This approach lets the organization as a whole balance speed and risk with stability effectively. Paying attention to this economy encourages investment in strategies that accelerate the business while minimizing risk: writing error- and chaos-tolerant apps, automating away pointless toil, advancing by means of small changes and evaluating “canary” deployments before proceeding with full releases.     

    Monitoring systems are key to making this whole, elegant tranche of DevOps/SRE discipline work. It’s important to note (because, remember: Google isn’t running your data center) that this has nothing to do with what kind of technologies you’re monitoring, the processes you’re wrangling or the specific techniques you might apply to stay above your SLOs. In short, it makes just as much sense to apply SRE metrics discipline to conventional enterprise systems as it does to 12-factor apps running on container orchestration. 

    Applying Google SRE Principles

    So here are a few things Google SRE can tell you about monitoring, specifically: 

    Alert only on failure, or on incipient failure – Alert exhaustion is a real thing, and “paging a human is an expensive use of an employee’s time.” 

    Monitoring is a significant engineering endeavor – Google SRE teams with a dozen or so members typically employ one or two monitoring specialists. But they don’t busy these experts by having them stare at realtimecharts and graphs to spot problems: that’s a kind of work SREs call “toil”—they think it’s ineffective and they know it doesn’t scale. 

    Post-hoc analysis, no magic – Google SREs like simple, fast monitoring systems that help them quickly figure out why problems occurred, after they occurred. They don’t trust magic solutions that try to automate root-cause analysis, and they try to keep alerting rules in general as simple as possible, without complex dependency hierarchies, except for (rare) parts of their systems that are in very stable, unambiguous states (their example of “stable”: when they’ve redirected end-user traffic away from a downed data center, systems can stop reporting on that data center’s latency). Elsewhere, their systems are in constant flux, which causes complex rule-sets to produce excessive alerts. One exception to this general rule about simplicity: Google SREs do build alerts that react to anomalous patterns in end-user request rates, since these affect usability and/or reflect external dependency failures (e.g., carrier failures). 

    Heavy use of “white box” monitoring – Google likes to perform deeply introspective monitoring of target systems grouped by application (called business service monitoring in Opsview Monitor). Viewing related metrics from all systems (e.g., databases, web servers) supporting an application lets them identify root causes with less ambiguity (e.g., is the database really slow, or is there a problem on the network link between the DB and the web host?) 

    Four golden signals – Because part of the point of monitoring is communication, Google SREs strongly favor building SLOs (and SLAs) on small groups of related, easily understood SLI metrics. As has been widely discussed, they believe that measuring “four golden signals”—latency, traffic/demand, errors and saturation—can pinpoint most problems, even in complex systems such as carrier orchestrators with limited workload visibility. It’s important to note, however, that this austere schematic doesn’t automatically confer simplicity, as some monitoring makers have suggested. Google notes that “errors” are intrinsically hugely diverse and range from easy to almost impossible to trap, and “saturation” often depends on monitoring constrained resources (e.g., CPU capacity, RAM, etc.) and carefully testing hypotheses about the levels at which utilization becomes problematic. 

    Conclusion

    The bottom line is that good DevOps monitoring systems need to be more than do-it-yourself toolkits. Though flexibility and configurability are important, more so is the ability of a mature monitoring solution to offer distilled operational intelligence about specific systems and services under observation, along with the ability to group and visualize these systems collectively, as business services. 

    — John Jainschigg

  • Why One Second Matters for Today’s Application-Centric Businesses

    Why One Second Matters for Today’s Application-Centric Businesses

    A lot can happen in a second, especially if your business is digital. Google receives 68,000 search requests. Amazon averages $2,361 in sales. In this application-centric world, those are just examples of what application outages can cost the business. Imagine if you didn’t even know there was a problem for just one minute. Sixty seconds of downtime would result in almost 3 million unfulfilled Google search requests and it would cost Amazon about $141,660 in missed sales.

    In an economy where consumers will abandon an application in six seconds if the experience isn’t satisfying, shopping carts are routinely deserted for poor performance and brand loyalty depends on immediate gratification, one minute of poor application performance can be disastrous for today’s digitally transformed organizations. Businesses must be able to see how their application (which is essentially their business) performs in real time, by the second.

    The Need for Speed in Today’s Modern Applications

    Those who code fastest will be the winners in a digitally transformed world. To increase velocity, developers are embracing more modern development methodologies such as Agile and using containers and microservices for their flexibility, scalability and speed. This is having a tremendous effect on the technology makeup of today’s applications and services.

    Operations must respond to this development speed revolution, especially when applications are deployed in a containerized environment. Consider this: Within 15 seconds of starting a script, an array of newly created containers can alter (or significantly impact) performance and/or quality of an organization’s applications—and its business. If those applications aren’t monitored within the same realm of speed at which the application is morphing, IT Ops won’t know about problems until unhappy customers start calling—something every business wants to avoid.

    Traditional application management products architected for monolithic applications struggle with visibility into modern applications. They also struggle to match the speed of change within the modern dynamic applications that digital businesses are rolling out today. In essence, using a traditional tool to monitor modern applications removes the benefits of a high-velocity development process through the increased risk of insufficient monitoring. High-resolution, time-based metrics (i.e., one-second granularity) are critical to modern application performance management.

    This need for one-second visibility actually goes beyond monitoring. Operations teams require feedback whenever a new application component is deployed, whether it’s a long-term server, a short-term container or a change of host. With application changes occurring with increased frequency and regularity, feedback is necessary in seconds, not minutes.

    One Second Means Business Success or Failure

    For monitoring tools to catch up with development progress and innovation, one-second visibility, or granularity, is essential.

    In today’s modern, dynamic application world, where the application is the business and possibly the only direct engagement with customers, one second means everything. It can mean the difference between business success or failure.

    The technology that allows developers to deploy applications so quickly is a great advantage, but that same attribute—speed—creates a huge hazard if these more complex applications can’t be adequately monitored and managed. The only way to ensure your IT Ops team—and your business—are protected against application outages is to monitor (and manage) your applications on a second-to-second basis.

    About the Author / Mirko Novakovic

    Mirko Novakovic is CEO and co-founder of Instana, where he brings deep technological experience with a background that includes performance management software, agile development and DevOps. Prior to Instana, Novakovic founded codecentric AG, a leading software consulting firm focused on performance, agile software development and software architecture. He received a Bachelor of Applied Science in Information Technology at Stuttgart University and an SMB Management at the University of St. Gallen. Connect with him on LinkedIn and follow him on Twitter.

  • SolarWinds Aims to Democratize Monitoring

    SolarWinds Aims to Democratize Monitoring

    In a move that has the potential to significantly expand the number of organizations embracing DevOps, SolarWinds announced it has expanded its IT management cloud service to include an integrated application and infrastructure monitoring offering dubbed AppOptics. Pricing for AppOptics starts at $7.50 per host per month.

    In addition, SolarWinds has updated Papertrail log management software to make searches faster and provide new visualization tools and the ability to resolve more complex problems such as application server errors and slow database queries. The company also updated Pingdom, a monitoring tool that can now provide views specific to a user, website or specific performance issues.

    Christoph Pfister, executive vice president of products for SolarWinds, says AppOptics should be make it much less expensive for organizations to embrace DevOps. Many organizations get caught up in a chicken-and-egg debate over whether to acquire tools before developing processes or vice versa. AppOptics renders that issue largely moot by making the cost of employing an integrated set of tools for monitoring infrastructure and applications nominal, says Pfister. That level of integration is now required to address digital business initiatives where the focus is more on the outcome than any one IT event, adds Pfister.

    Pfister says SolarWinds is the only provider of a software-as-a-service (SaaS)-based set of monitoring toils that support multiple programming languages, which he notes creates yet another opportunity for DevOps teams to reduce costs.

    AppOptics includes support for auto-instrumentation, distributed tracing and host agent software. That tracing software is based on technology SolarWinds gained when it acquired Librato. Adding instrumentation to any application can be accomplished using a single line of script, says Pfister. The host agent support is achieved using Snap and Telegraf plug-ins to monitor systems.

    By making it easier to add instrumentation, Pfister says SolarWinds expects the number of applications actively being monitored will increase substantially. Historically, only a small percentage of applications deployed in an enterprise are actively monitored. Pfister notes that a much higher percentage of new applications being developed are “systems of engagement,” which require much higher levels of instrumentation to gauge acceptance by end users, and not “systems of record.” Gartner says that only 5 percent of organizations today have implemented digital experience monitoring (DEM), a number it forecasts will rise to 30 percent by 2020.

    At a $7.50 per host per month price point, AppOptics should go a long way to commoditizing DevOps tools. That should not only spur more adoption, but also could force a wave of consolidation across a crowded category of monitoring tools as profit margins get squeezed. In many instances, it’s also likely that monitoring increasingly will be bundled with other tools and services as vendors aim to compensate for monitoring becoming a commodity by selling it alongside higher margin products and services.

    Long term, it’s apparent that cost is increasingly being removed as a factor in the DevOps equation. Not every organization is going to able to master every DevOps tool available. But when it comes to monitoring, going forward there should be no financial excuse for failing to have at the very least some minimal level of DevOps capability.

    — Mike Vizard

  • CA Technologies Extends DevOps Push

    CA Technologies Extends DevOps Push

    At its CA World 2017 conference this week, CA Technologies moved to tighten the integration between the various elements of its DevOps portfolio acquired over the last year and unveiled CA Digital Experience Insights, an analytics-infused monitoring tool delivered via a software-as-a-service (SaaS) model.

    The company also announced an instance of the CA Microservices Gateway that allows the API management platform to be deployed as a Docker container. Earlier this week, CA Technologies launched a beta release of Fastrack.io, a service for monitoring Kubernetes clusters based on machine learning algorithms and embedded in an instance of the open-source Prometheus container monitoring service.

    The company is also adding CA Virtual Test Data Manager, a tool that makes it simpler to create copies of large data sets for testing purposes, and CA Service Virtualization Community Edition and CodeSV, a plug-in that enables DevOps teams to invoke service virtualization from within their integrated development environment to mimic external services without having to connect to an actual instance of those services running in a production environment.

    Additionally, CA Technologies announced it has integrated CA Veracode security testing software with CA Automic IT automation platform, and announced general availability of BlazeMeter API Test, a SaaS-based API testing tool, and CA Continuous Delivery Director SaaS, a service that integrates the DevOps toolchain.

    Aruna Ravichandran, vice president of DevOps product and solutions marketing for CA Technologies, says each of these offerings are part of a larger company effort designed to enable organizations to set up their own software factories. Now that more organizations have come to realize their ability to differentiate themselves depends on custom software, many of them want to be able to acquire all the integrated DevOps tools required to achieve that goal from a single vendor, Ravichandran says.

    CA Technologies is making a concerted effort to link the need to create a software factory as a prerequisite for driving digital business transformation at scale. To achieve that goal, she says, control over IT operations is continuing to shift to the left as developers become more accountable for the entire application experience. Offerings such as CA Digital Experience Insights make it simpler for developers to holistically manage those end user experiences by infusing predictive analytics and machine learning algorithms coupled with improved data acquisition techniques in a single service, Ravichandran says.

    CA Technologies clearly wants to become a DevOps force to be reckoned with. In the last two years it has acquired Blazemeter, Automic, Veracode and Runscope. Prior to that, the company acquired Rally Software, Nolio, Grid Tools and Layer 7 Technologies. From here it’s likely CA Technologies will continue to expand its portfolio by acquiring companies while tightening the integration between the various code bases that make up its DevOps portfolio.

    The company’s stated goal is to provide a suite of DevOps tools and services that are too compelling to ignore without forcing IT organizations to consume a product or service they don’t want by maintaining open interfaces across the portfolio. The degree to which CA Technologies can succeed in that effort remains to be seen. But there’s no getting away from the fact that the company has now assembled a formidable DevOps arsenal of products and services.

    — Mike Vizard

  • Top 5 Best Practices for DevOps Monitoring

    Top 5 Best Practices for DevOps Monitoring

    Today organizations are required to deliver on higher levels of customer satisfaction for their online services. Many, however, are forced to support these initiatives with an interrupt-driven approach as they react to fix things after they break. However, for a more proactive approach and to manage expected high levels of SLAs, organizations can opt to reduce their amount of unscheduled downtime by implementing a continuous delivery (CD) model to their development efforts.

    What is critical in the CD model is the ability to monitor and manage systems in a structured way, getting early detection of problems so organizations can make changes before the system is impacted. While unexpected failures make interrupt-driven work unavoidable at times, organizations can become more proactive by examining their current approach and toolset against business needs to help create a path to continuous service delivery optimization.

    Effective CD Models Need Effective Monitoring

    As a starting point, organizations need control and visibility into their DevOps environment by collecting and instrumenting everything. Considering the amounts of data, this can be an insurmountable challenge for most organizations. To get started, follow these five best practices to perform DevOps monitoring efficiently and quickly in a measurable and scalable way.

    Step 1: Collect the Data. You can’t manage what you don’t measure! It is important to take inventory of what is being collected today and align with business and executive teams to collaborate on the aim of the services delivered. Analyze the inventory collection of metrics with questions such as: “Why aren’t we collecting this? How does this fit into our goals? Have you ever seen a failure in X? How often should we be measuring this? How long should we keep it for? Is that important?” Teams also should evaluate how they are collecting the information and consider the best architectural approach for collecting their metrics, including whether pull, push or pull collection methods are better. Once goals are understood and the inventory of data is collected, look at what else your organization should consider collecting.

    Step 2: Correlate and Triage. Correlation of data is needed to understand it, but data comes in at different frequencies, in different time frames and from different sources. Work to normalize the data to understand it, perform comparisons against the different incoming metrics and establish baselines for basic service availability. Since organizations aim to go beyond basic service level agreements (SLAs) and offer a high-performing solution, constantly ask what the organization is missing from a data perspective and how it relates to business initiatives. Asking that question from the collection, correlation and triage perspective is critical.

    Step 3: Identify Trends. Organizations need to examine historical data to identify trends and take action before issues arise and customers complain. Establish alerting thresholds by outlining what a normal day looks like from a monitoring performance and customer perspective, and then identifying examples of what makes an abnormal day. This ties in with managing infrastructure inventory and understanding safety thresholds for each of the components that potentially could impact the service offering. It’s critical to communicate these findings with teams and business-line managers to prevent service delivery problems from happening and optimize offerings based on identified trends.

    Step 4: Notify and Act (Automation). In manual mode, a notification is delivered and then the team reacts. But teams are continually pushed to do things faster, and automation can help. To get to that point, organizations must understand where best to add more automation. How do you gather the right telemetry that delivers consistent answers that the machine can operate rules against, and do you leverage an automated process or notify a person? The desire to improve to a faster process requires a shift from manual to automated practices.

    Step 5: Predict (What-if Analysis). If you don’t take the time to go through the first four steps in a methodical way, it’s very difficult to reach this final step without always remaining in a manual mode. To balance costs and availability, it’s critical to discuss with the executive team how to predict customer service consumption (revenue) against the amount your services are going to cost the business.

    For example, developing a service that can alert customers of a potential service disruption due to disk space can be accomplished only by clearly setting business goals, then using the right metrics and events platform leveraging a time series database. In this case, the business goal is to ensure no service disruption due to failures such as low disk space; the metric monitoring is for disk space for each customer instance with the appropriate threshold alert set off by an automated trigger; and the action is an email to the customer letting them know of the situation and what action they can take (reduce load or upgrade for more disk space). In this example, it is important to marry business logic with the monitoring practice to make service a successful experience for customers. Furthermore, this approach will help teams to not only predict user experience but also help to predict OpEx and CapEx better into the future.

    That’s the benefit of the ability to predict with the right monitoring approach—to make informed business decisions based on results. Customers appreciate that, and proactive support is much better than being reactive.

    Emerging trends such as microservices, containerization, elastic storage, software defined networking and hybrid clouds are pushing the boundaries of DevOps monitoring. The right monitoring plan can identify and resolve problems before they affect critical business processes and enables customers to plan for upgrades before outdated systems begin to cause failures and outages for users.

    About the Author / Mark Herring

    Mark Herring is CMO at InfluxData. He is a well-rounded silicon-valley executive with proven experience in taking complex technology and making it understandable to the broader audience. His deep developer roots are never far from his mind as he looks at trends and asks the tough questions on whether the technology is here to stay or just another fad. Currently Mark is Chief Marketing Officer at InfluxData. Follow him on Twitter and connect with him on LinkedIn.