devopsy.kinsta.cloud

Category: AI

  • AppDynamics Applies AIOps to Automate IT

    AppDynamics Applies AIOps to Automate IT

    AppDynamics, a unit of Cisco, today announced it is melding the worlds of AIOps and application performance management (APM) with the launch of Cognition Engine.

    At the same time, AppDynamics is making good on a promise to integrate metrics drawn from Cisco Application Centric Infrastructure (ACI), the software-defined networking (SDN) that Cisco is embedding in all its networking equipment.

    Finally, AppDynamics also announced Serverless Agent for AWS Lambda, which marks the first time the company’s APM platform has been extended to include the ability to capture metrics from every single invocation of a serverless computing framework.

    Matt Chotin, senior director of technology strategy at AppDynamics, said Cognition Engine represents an ambitious effort based on machine learning algorithms and predictive analytics to create a central nervous system for IT spanning applications, infrastructure and the network. The days when applications and infrastructure were managed in isolation are now coming to an end, he said.

    Available at no extra charge to AppDynamics customers, Cognition Engine infuses the Business Transaction data model underpinning the AppDynamics APM services with machine learning algorithms to run diagnostics and automate root-cause analysis in a way that reduces resolution times from minutes to seconds. The machine learning algorithms evolve constantly as data is gathered continuously using streaming analytics to ingest, processes and analyze millions of records per second. The Cognition Engine automatically understands how metrics correlate and instantly knows when there’s been a break in correlation among datasets.

    Cognition Engine also automatically isolates metrics that deviate from normal behavior to identify the top suspects of root cause for any application issue. As an open platform, that information then can be shared with orchestration and automation systems embedded within third-party incident response, intelligent workload placement and cloud cost optimization tools and platforms.

    These types of capabilities are now becoming an absolute requirement because IT in general is becoming too large and dynamic to manage relying on manual processes, said Chotin, adding what’s needed is a shift toward an AIOps mindset that automates the management of routine IT tasks at scale.

    AIOps is, of course, not so much a thing that can be purchased as much as it is a collection of technologies spanning machine learning algorithms, predictive analytics and other technologies that are employed in unison to automate IT operations. It usually takes a while for algorithms to learn a specific IT environment. But once they do, they bring with them the benefits of a more sophisticated approach to managing IT at scale (and they never take a sick day or decide to take another job). Of course, AIOps is not likely to replace the need for IT staff entirely. But roles and functions within IT organizations most certainly will evolve in the months and years ahead.

    In the meantime, organizations will increasingly vet AI models before deploying them in production environments. While AIOps will advance the state of IT as it is known today, the most worrisome thing about any form of current AI technology is being catastrophically wrong at a level of scale that is nearly impossible to recover from.

    — Mike Vizard

  • AIOps: How to Solve Performance Issues in Complex Data Pipelines

    AIOps: How to Solve Performance Issues in Complex Data Pipelines

    AIOps takes center stage to topple barriers as modern data application growth accelerates

    There seems to be a chasm between DevOps and big data teams in the enterprise, and the bridge over that chasm is long and treacherous.

    In fact, data analysts who are charged with managing and making sense of the growing volumes of data in most organizations traditionally have been excluded from the strategy-making and operations functions of DevOps personnel. Yet, data analysts are critical to the success of software development and at every stage of the software delivery pipeline.

    DevOps Under Siege as Modern Data Applications Proliferate

    Let’s take a deeper look. We know that DevOps personnel are on the front lines in ensuring that all systems, both new and existing, perform as expected in a production environment. But, even today, DevOps teams struggle in dealing with technical challenges when faced with complex data pipelines and the underlying big data stack that supports them, including:

    • Root cause analysis — determining if an application issue is due to application code, config errors or resource contention.
    • Correcting inefficiency in code.
    • Improperly configured auto-scale clusters.
    • Lack of actionable insight into how clusters were tuned at both a cluster and job level.
    • Difficult access to support engineers and promotion of DevOps collaboration.

    And these are just the tip of the iceberg.

    The pressure is on. On average, companies with data-driven environments garner up to 50 percent higher market value than their competitors, according to IDC, and big data is a contributing factor. Thus, big data implementations have become a top priority for businesses of all sizes, and on multiple fronts: revenue acceleration, creation of new products and contributing to better, quicker decision-making. But the unfortunate truth is that, as big data projects move from pilot to production, they invariably encounter serious performance problems that frustrate both data analysts and IT operations.

    What’s more, DevOps and big data personnel face a mutual problem: lack of centralized, top-down visibility into their end to end data pipelines and applications. And, as a result, they face challenges in pinpointing the source of problems—namely, that problems can become visible anywhere in the stack, but their root cause could be elsewhere in in the stack. And, these problems result from root causes as diverse as bad code, inefficient data partitioning, mismatched system configuration settings, best-guess resource allocation and infrastructure issues. Furthermore, a big data stack running multiple applications faces resource contention and prioritization issues that can reduce both individual application performance and overall system performance.  And streaming applications create their own challenges;: for example, if a streaming application starts to lag in processing data in real-time, diagnosing the root cause takes considerable time and effort.

    The industry is faced with a reality where the problems are outrunning the solutions. In the case at hand, organizations have relied on brute force and expensive custom solutions for making analytics-based recommendations more user-friendly to improve individual application and full-stack performance.

    AIOps Takes Aim

    AIOps, or artificial intelligence for IT operations, is gaining steam in the enterprise, with many new use cases appearing as data volumes grow and systems become more complicated. Gartner expects AI to pervade every aspect of managing IT, including application performance management (APM), IT infrastructure monitoring, cloud platform monitoring, performance management, capacity management, the delivery tool chain (CI/CD) and IT service management.

    In fact, AIOps is on a tear: Just three years ago, it was only a concept. Today, Gartner predicts that by 2019, 25 percent of global enterprises will have implemented an AIOps platform to support two or more major IT operations.

    But even as it continues to evolve, AIOps today can deliver automated recommendations to solve expensive, challenging and time-consuming problems with big data deployments and across entire complex stacks. Without AIOps, there would be simply too many variables to correlate manually.

    AIOps makes APM for modern data application possible. It does so by helping to eliminate painstaking, trial-and-error configuration, troubleshooting and remediation workflows that have long plagued big data operations teams. And, for the first time ever, dataOps and DevOps teams can troubleshoot, support SLAs, remediate issues and manage large-scale deployments in an agile and dependable manner.

    It’s a new environment for the modern data center. As part of an APM solution, AIOps is hard at work helping automate and guarantee the performance of big data applications and addressing the primary challenges that DevOps faces with the unrestrained growth of big data.

    It could still be a bumpy ride—organizations of all kinds face unprecedented increases in data volumes. But now, working together, DevOps and AIOps are better-equipped to contend with the challenges.

    — Bala Venkatrao

  • How AIOps Makes DevOps Less Noisy

    How AIOps Makes DevOps Less Noisy

    For DevOps engineers, “noise” is the enemy of productivity. In this context, the noise we’re talking about is unnecessary or low-priority alerts and notifications that distract engineers from identifying serious issues—and ultimately can cause alert fatigue syndrome, in which alerting systems are ignored altogether.

    Without the application of a well-constructed noise reduction plan, alert noise will end up taking away from the focus of solving the actual problem that may be causing a more critical issue.

    So, how can you fight alert noise? You might be tempted to assume that the answer lies in reducing the number of alerting tools you use, but that’s probably unrealistic. Teams today are dealing with complex application stacks and multiple tools and data sources, all of which can produce their own alerts for issues that may be interdependent or isolated. You simply can’t take tools out of your stack.

    What you can do, however, is analyze and manage alerts more effectively. Dealing with alert noise ultimately boils down to developing an effective problem analysis strategy. Let’s explore how, with a specific focus on how AIOps factors into keeping alerts manageable.

    Reduce the Time in Analyzing Alerts

    While reducing noise is important, DevOps teams risk losing valuable time solving critical problems if they also have to create and/or add systems to properly analyze alerts and their sources and priority level. Maintaining this system also would be a source of unnecessary labor that may reduce the alert noise, but also reduces the amount of time that DevOps staff can work on the solution to the root cause— which brings up another consideration. When coming up with a plan, the amount of complexity involved in adding the monitoring tools necessary to cover an entire application and/or service can be equally laborious compared to not having a solution at all. At this point, we may be merely shifting the labor emphasis.

    This is Where AIOps Comes In

    Now, let us envision a way that a team can gain full insight into a root cause without exposure to alert noise. The team doesn’t need to analyze every single alert, which would include:

    • Duplicate alerts.
    • Alerts from multiple sources (tools, microservices).
    • Informational alerts (warnings).

    Instead, the team can let AIOps analyze and classify the stream of alerts and identify critical issues. DevOps no longer would need to spend valuable collaboration time analyzing every link in the toolchain. When a team has to analyze every tool, the team could produce false positives, redirect resources toward the wrong solution, spend too much time trying to identify the root cause and create excessive alert noise. Let us envision a way in which the team does not need to spend excessive time in analysis and can experience more effective collaboration in service of solving the actual problem.

    AI and machine learning can relieve this manual process through AIOps. Automated collection of data across the application stack, alert correlation, identifying root cause and decision-making can be leveraged by a team to get to the problem immediately without all of the messy work of manual analysis, sorting and identifying root cause. This level of automation allows for needed human interaction through collaboration across teams, which can result in faster resolution, decreased downtime and effective communication of key events to company staff and end users.

    Analyzing to Predict Future Problems

    Another benefit of AIOps and the power of automated data collection, analysis, issue identification and decision-making is the ability to predict future events. A DevOps team can leverage these insights into effective preventative measures. The ability of AIOps to produce these insights will help a team find ways to automate and/or improve operations. Reducing alert noise no longer needs to be a goal because it is happening through a smooth interaction between the human team and the AI.

    There is a growing need for DevOps teams to utilize the power of AIOps. While human teams are not replaceable, redundant and time-consuming tasks are. The ability of AIOps to effectively traverse a complex toolchain with isolated issues to analyze alert importance, correlate alerts, collect and analyze relevant data, identify root cause and produce holistic actionable insights eliminates inefficiencies in problem-solving and reduces alert noise.

    Making DevOps less noisy is a goal with its roots in effective use of information. Alerts serve to inform, but thanks to the increasing complexity of applications, alerts become noise. Leveraging the power of AIOps serves to do a lot of the dirty work for a team so that noise can be reduced—but more importantly, it can make operations run smooth and keep customers happy.

    — Phillip Lorenzo

  • How AIOps Helps You Get More Out of DevOps

    How AIOps Helps You Get More Out of DevOps

    DevOps may seem like an overused term. It involves taking an application’s source code and running it in an environment. It can cover the processes, the technology or even the people that maintain the technology that is running those very processes.

    At its heart, though, DevOps is about helping developers to be self-sufficient when it comes to the basic operations around getting their application to a real environment. Once things are running in a test or production environment, there are other IT functions that are required to keep it running and performing well. While there are teams that specialize in these functions, such as site reliability engineers (SREs) and system administration, they are usually overwhelmed by the sheer amount of data their areas of responsibility generate without considering the data that applications going through their DevOps processes are also creating.

    This is what makes AIOps the perfect complement to the DevOps space.

    What AIOps Does

    What AIOps does best is break down silos within IT. Having data from all the functional areas processed and handled by modern machine learning (ML) in a single place allows correlation, which leads to faster problem identification and resolution.

    Before machine learning was introduced, for humans to process data, more or less they operated in three ways:

    • They knew exactly what data to look for, such as when a specific business function isn’t working. There is a typical message in the monolithic application that handles all the core business.
    • They routinely sifted through data only from their area of expertise.
    • When they went outside their area, any logs and other data they had access to often was summarized data, which has time gaps and can miss a lot of detail that proper correlation needs.

    Now with ML, the AIOps data platform can perform trend analysis, look for patterns and exceptions across all the data that is being generated in the environment and highlight what it considers important through web interfaces or using an event management solution for after-hours support.

    Over time, as the AIOps platform continues to learn the environment, it can be given the ability to automatically resolve certain error conditions, as it will be able to identify them faster than human operators and often resolve them before they become service-impacting.

    How This Helps DevOps

    Machine learning aids DevOps processes in three ways when working with logs and data:

    • Knowing exactly what data to look for: In a true polyglot DevOps environment that is leveraging microservices, it is next to impossible for any one person to know the most common error messages that might occur. In this model where a new business function can be conceived, written in one of many languages, and deployed into production without Operations even being notified, ML allows the AIOps platform to quickly identify what “normal” looks like for a new microservice and start looking for anomalies.
    • Only routinely sifting through data from their area of expertise: When working in DevOps, developers need to be able to quickly find data from multiple areas in IT that is related to whatever they are trying to diagnose. Using AIOps, they will be able to quickly pull data from systems directly related to their incident. Whether it is as simple as using timecodes or a correlation GUID (globally unique identifier) that is put in the logs purposefully to aide tracking, AIOps will guide the developer through the other systems’ data.On the flip side, if there is an infrastructure problem, the same AIOps platform can help correlate what downstream services may have been impacted, from failed builds to Kubernetes pods that were restarted.
    • Summarized and incomplete data: The true value of machine learning is that it has the ability to process a higher volume and variety of data at a much higher speed and accuracy than a human, and sort and index it for later use. Not having to summarize the data to one data point every five minutes could result in a problem being discovered that has occurred often but not consistently, as it may have been dropped in the summarization.

    Conclusion

    AIOps is very young in the technology industry, but has great promise as it builds on previous generations of IT operational analytics products, combined with the latest in machine learning capabilities. DevOps is a key field in IT that has more logs than it knows what to do with. Combining the technologies and expertise in these two growth areas will have a huge positive impact on productivity across enterprises with improved service management.

    — Vince Power

  • DevOps Chat: AIOps and APM with Unravel’s Kunal Agarwal

    DevOps Chat: AIOps and APM with Unravel’s Kunal Agarwal

    Unravel is making a name in the AIOps, APM and big data markets. In this DevOps Chat, we speak with Unravel CEO Kunal Agarwal about what makes Unravel a must-have for its customers. We also take a sneak peek ahead to what the near and not so near future holds for big data, AIOps and the tech scene in general.

    As usual, the streaming audio is immediately below, followed by the transcript of our conversation.

    Transcript

    Alan Shimel: Hey, everyone, this is Alan Shimel, devopsy.kinsta.cloud, and you’re listening to another DevOps Chat. Today’s DevOps Chat features Kunal Agarwal, CEO of Unravel or Unravel Data. Kunal, welcome to DevOps Chat.

    Kunal Agarwal: Thank you, Alan. Thank you so much for having me here.

    Shimel: Excellent. It’s my pleasure to have you here. So, Kunal, let’s get right to it. “Unravel” is the name of the company, correct? Not “Unravel Data.”

    Agarwal: “Unravel,” yes.

    Shimel: But yet Unravel is about data to a certain degree. I don’t know if everyone in our audience is familiar with Unravel. Why don’t we start there?

    Agarwal: Sure. We are unraveling the mysteries of big data, Alan. That’s why the name of the company.

    Shimel: Excellent.

    Agarwal: Everybody’s jumping on to big data apps, and what I mean by that is you may be doing something with machine learning, artificial intelligence, IoT, even big-scale data warehousing. And there’s just so many problems that can happen, running these applications at scale, that people just have not seen before. And this is a problem because of all these new systems and technologies that are coming about, such as Spark, Kafka, Hadoop, Cassandra, HBase, many, many more. And the complexity’s all about “Hey, how do I figure out why’s my application not performing well or if it’s failing?” And it’s usually a multi-tenant environment that these big data applications run on, which means there may be hundreds of users, thousands of applications, that you are creating every day and running them.

    And figuring it all out and making sure that they run well is a black art, so there’s a lot of trial-and-error that goes on into fixing these problems, finding the root cause of these problems. And that’s what we are automating, meaning Unravel, after it’s installed in your big data cluster, will be constantly monitoring and managing everything. And if there is an issue, Unravel will be proactive about it, tell you why that issue happened, what caused it, and then even take a step further where it would resolve those issues for you so that you can spend time creating all these amazing big data applications, rather than focusing on fixing all these systems- and application-level errors.

    Shimel: Mm-hmm. So, you know, I mean, there’s a lot there that we’re gonna dive into, but before we do, Kunal, just give our audience – ’cause I always find this fascinating. People wanna know “How do I become a CEO? How does one get to be the CEO?” Give us a little insight into your own personal journey on how you’ve come to be CEO at Unravel.

    Agarwal: CEO is “chief everything officer,” when you first start off. You gotta sweep the floors and you gotta raise the funds and you’ve gotta grow a team. There’s so many things a CEO’s role requires to do, especially in a young company, as you’re growing it. Today, now that we’ve grown to about 75, 80 people in total, across 3 different countries – we actually have offices here in the United States, one in London, and one in India – my job is becoming more about recruiting, more about getting the team together.

    And I think that’s, personally, for me, if that is taken care of, the entire company just hums and does well. Everything else is secondary for me at this time, but I do spend a lot of time doing marketing and sales and finance and product. So it really depends on the day, Alan, of what my job requires me to do.

    Shimel: Amen. You know, I’ve founded or cofounded three or four companies myself and even just running MediaOps, which is devopsy.kinsta.cloud and Security Boulevard, you are chief cook and bottle washer when you’re – you’re chief everything officer, as you said.

    Agarwal: Right.

    Shimel: You know, looking in your background, you’re a double-major graduate – electrical engineering, computer science master’s degree from Duke. I mean, you’re well positioned to be a CEO, people would say, looking at the background, but it’s – you know, you never know what’s coming out next as the CEO. If I had to ask you, Kunal, what’s kind of been the biggest surprise or biggest kind of curveball that you’ve seen in your time at Unravel that you think, “Geez, I didn’t think I’d be doing this or I didn’t think this was part of the job,” what would you say?

    Agarwal: Alan, actually, going to school and getting all those double-degrees does not help you at all becoming CEO.

    Shimel: You get it. I mean, you’re talking to someone who went to law school. I know all about it.

    Agarwal: Right?

    Shimel: Yeah.

    Agarwal: I wish there was a school for CEO, but you know what? The only school for being CEO and leading a company is actually leading the company because there’s so many things that you learn along the way. One thing, because we deal in enterprise software, there’s so many complexities to making a deal happen with a big company.

    Shimel: Mm-hmm.

    Agarwal: And I think that was one of the things that took me a while to grasp and make sure that we’re doing it right. And I mean all the boring stuff, like the contractual paperwork, all the legalities, going through info-sec requirements. All the operational stuff around making a deal happen. You know, when you’re in college and you’re starting to think about a startup, you think about the cool products you’ll make. You’ll think about the impact your product will have in the market and how so easily everybody will flock and come to you and get your product. [Chuckles] But it’s not that simple.

    Shimel: I wish it was.

    Agarwal: and it takes 9 months to 12 months to close some of these deals, so I think, more than the “surprise” factor, preparing yourself and making sure that you can drive efficiencies in all these different processes. I think that’s been one of the key challenges for us.

    Shimel: Absolutely. Absolutely. So, Kunal, if I’m a Silicon Valley VC, I’m looking at a company like Unravel, man, you’re playing in all the right places. Big data is certainly such a huge – I think, when we talk about things like machine learning and AIOps, really, they’re almost the children of big data – you couldn’t have those things without the big data, first of all – so you’re in the right place, obviously. The right time. This is a very happening kind of space. What is Unravel doing that is so – what’s the special sauce? What’s unique about it?

    Agarwal: Yeah, Alan. So, when we created this company – and by “we,” I mean myself and my cofounder, Dr. Shivnath Babu, who’s actually a professor of computer science at Duke University – we both met at Duke – and the problem we were trying to resolve is one of the most fundamental but also probably the most complex and challenging problem to solve, which is “How does one ensure that all of these big data technologies actually work and deliver on their promise?” And, you know, our approach to the problem was always “Hey, there will be a shortage of experts in the market. Even if there are experts, these problems will be requiring a machine to solve them on time and efficiently.”

    So we took the long path of actually creating not just a monitoring tool but a AIOps tool, if you may, which can decipher the problem, connect the dots, and resolve it for you. So it actually took us about three-and-a-half years to build this product or at least the first version of this product. And we moved west to start working with some companies – I lived in New York; he was in North Carolina – and we gave up our cushy jobs, moved to Silicon Valley, because you wanted to work with companies who were using big data tech at scale.

    And some of these companies, we still have advisors in, from Zynga, Rocket Fuel, Twitter, LinkedIn, and they really helped us understand the day-to-day issues that come up when you’re running these technologies at scale. And we took a lot of those learnings into our first product. So we were always about solving a core need that teams in big data tech would definitely need. And, you know, they always say, “Hindsight is 20/20,” right?

    Shimel: Yeah.

    Agarwal: But we always put a big bet that, “Hey, these technologies – Hadoop, Spark, Kafka, et cetera – will become mainstream and companies will start using this a lot more.” So what LinkedIn was doing five years ago, the biggest banks would start doing it now, and that’s where we are in the journey today.

    Shimel: Absolutely. And, you know, it’s funny, full circle, talking about that journey and what you’re doing. That’s how you become a CEO. It’s about being willing to take the risk, giving up those cushy jobs. Right? To do something that you really believe in. And I think that’s what separates CEOs from people who kinda look and say, “Boy, I’d like to do that,” but they don’t actually do it.

    Let’s talk about the market. We’re coming to the end of the year. You’re participating in our Predict 2019 panel, which will be December 18th event. You’re gonna be on the AIOps panel, looking ahead. Why don’t you give us a sneak peek? What’s kind of your take on what 2019 holds for AIOps and big data and so forth, Kunal?

    Agarwal: Well, yeah, Alan. So AIOps is an evolution of IT Ops or APM or DevOps, where you’re using machine learning to, again, understand all of the data – and this could be monitoring data, instrumentation data – that all these systems and applications are generating to draw insights from.

    Now, funnily enough, this is what our customers do with their data set, right? So with some of the biggest banks, some of the biggest insurance companies, some of the biggest health care companies, all Fortune 500. And our methodology of going about monitoring for big data stack was always the same, which is “Why aren’t we using big data technology on our data set, which is machine-generated data, to glean those insights and drive some prediction and actually make things simpler and more accurate?”

    So AIOps is also one of the use cases of artificial intelligence and machine learning. Now, just as I’ve said the industry’s evolving, what we predict to see is the evolution of AIOps, so Unravel, for example, is already on one of the end-of-the-spectrum places, where we actually make decision on behalf of the operations and the development teams, whether it is re-evaluating an application, whether it is making sure that your SLAs for your applications are being met, by moving low-priority jobs to a sub-optimized queue and high-priority jobs to a priority queue or killing bad applications in the cluster, all on your behalf, to make it more proactive.

    But what I predict is there’s gonna be an evolution in just the incumbent APM companies as well, where it’s not just about giving people a monitoring screen or even a native monitoring screen and saying, “Hey, check out all the information we know about an application,” but then I leave it up to you to decide and understand what’s amiss, what’s wrong, and how do you fix it. But I feel we’re already now in the correlation and the insights phase, which is now I can start to connect the dots, with these new software technologies that are out there. But, in 2019, we’ll start to also see some of these companies take the leap to be able to do automated root cause analysis, as well as guided remedies or automatic solutions, to get out of these problems as well.

    This industry’s evolving at a very fast pace, so I’m super excited. And it’s all about making sure that some of the biggest applications in the world, which millions of people actually rely on, all work like a well-oiled machine. And it’s all about guaranteeing reliability, which, at the end of the day, improves confidence of the customers using those products, which further fuels more innovation and more growth. So it’s an amazing cycle and AIOps is gonna be a very, very key part of it.

    Shimel: No doubt about it. No doubt about it. You know, in many ways, I look back on 30 years in IT and it really is a golden time to be alive, if you’re a developer or ops person or just in IT in general. Right? There’s the possibilities and the tools, and what’s in front of you is so exhilarating, right? I mean, I think back, if I was 20 years old now, 22 years old, coming into the –

    Agarwal: So you mean like 10 years ago, Alan?

    Shimel: Yeah, something like that. About 10.

    Agarwal: Okay.

    Shimel: About 10. And, I mean, just unlimited opportunity. I mean, that’s kinda almost the way I look at it. And, of course, it’s companies like Unravel, right, that are kinda letting these opportunities take shape and packaging them for people to run with. So great stuff. Specifically, Unravel – we only have a couple moments here left, but – so, Kunal, anything on the horizon with Unravel you wanna share with our audience?

    Agarwal: Yeah, Alan, absolutely. So, even in big data technologies, there’s so many to cover. We started our life off supporting Spark, Kafka, the entire Hadoop stack, some NoSQL systems, like HBase and Cassandra, and some newer NPP platforms. In 2019, we’re gonna further expand some of those technologies and systems that we cover, especially as we start to see more and more big data workloads move to the cloud.

    We’re also seeing a lot of our companies demanding a hybrid real estate management from the cliched “one pane of glass,” that “Can I see everything that’s in my big data environment, whether that’s on premise and on the cloud, together?” So Unravel is launching that support as well.

    And, last but not least, we are also creating a solution, from our product, which is directed towards helping companies move from their on-premise environments to cloud environment. So actually using Unravel as a migration project management tool, to help you assess, migrate, and then validate your migration. And, on the assess side, figuring out “What system should I get? Which instance type is best for my workloads? How can I keep under cost? What is the cost of running my big data workloads on the cloud?” to then validating and making sure that your SLAs are met, that you’re under cost and under time, and everything is flowing like a well-oiled machine.

    The way we look at it, Alan, again, the way to look at our road map and anything on the horizon is we are here to help our customers in the entire journey of big data tech, so, whatever they’re deploying today and they wanna use tomorrow, Unravel will be supporting that. And this is a natural evolution where companies are going from trying something out to getting something in production, first starting on premise and then going to the cloud, and those are all the shifts and transition points that we wanna hold the hands of the customers that we work with.

    Shimel: Excellent. Excellent. Well, Kunal, thanks for spending a few moments with us. As I told you, the time goes quickly.

    Agarwal: Yeah.

    Shimel: We’re way over time, but it’s okay. Hopefully, people have stayed on till now to listen. I wanna wish you lots of continued success with Unravel. We’ll be checking in on it. Again, a reminder: Kunal will be on our December 18th Predict 2019 virtual event and Kunal will be participating in the CEO panel, taking a look ahead at AIOps in 2019. So join us. You can register for that at predict2019.com.

    Agarwal: Thanks, Alan.

    Shimel: Thank you. Hello to all of our friends at Unravel. And that’s gonna make a wrap on this episode of DevOps Chat, everyone. This is Alan Shimel for DevOps Chat, devopsy.kinsta.cloud. Have a great day.

    — Alan Shimel

  • VMware Wavefront Adds AI, Distributed Tracing

    VMware Wavefront Adds AI, Distributed Tracing

    VMware has updated its Wavefront monitoring service to include support for distributed tracing along with anomaly detection capabilities enabled by machine learning algorithms.

    Stela Udovicic, senior director for product marketing for Wavefront by VMware, said the addition of the distributed tracing support, now available as part of a public beta, means that a single monitoring service now can provide metrics, histograms and the tracing function required to provide a holistic view of the IT environment. Those capabilities eliminate the need to employ multiple services to achieve the same goal, she noted.

    In contrast, Wavefront makes it easier to search for traces containing a specific service or application programming interface (API) to determine which ones have errors or long response time. Additionally, DevOps teams can filter the trace search to show traces containing spans from a given cluster or shard.

    At the same time, VMware is adding an AI Genie capability that promises to make it easier to troubleshoot existing needs and identify future application and IT infrastructure requirements, said Udovicic. That data is collected in real time and presented in a way that allows even the most novice users to visualize using analytics tools built into the Wavefront services, added Udovicic.

    Wavefront is based on an agentless architecture that in part makes it possible for the service to process millions of metrics, histograms and traces per second. That capability is especially critical in IT environments that have embraced microservices based on containers that tend to expand across an enterprise IT environment, said Udovicic. Wavefront already supports Kubernetes clusters.

    In general, Udovicic noted that developers tend to embrace a variety of open source approaches to monitoring applications as they build their applications. But a SaaS application such as Wavefront makes it easier for enterprise IT organizations to apply monitoring across a broad range of applications. To facilitate that transition, Wavefront provides an OpenTracing/OpenCensus compliant solution that includes a drop-in replacement for the Zipkin and Jaeger distributed tracing software, which is at the core of many open source monitoring tools.

    VMware is clearly betting that increased reliance on AI will push organizations toward relying more on cloud services. Machine learning algorithms require access to massive amounts of data before they can construct AI models capable of accurately predicting events. The cost of collecting and storing all that data, plus the expertise required to train the AI models, are beyond the capabilities of most DevOps teams.

    Most enterprise IT organizations have applied monitoring tools to only their most critical applications, mainly because of cost concerns. But as applications based on microservices proliferate across the enterprise, it’s becoming apparent that all the dependencies that will exist between various application services will need to be continuously monitored. The only issue now is determining how best to achieve that goal as part of an integrated set of DevOps processes.

    — Mike Vizard

  • Cognigo Secures $8.5 Million Series A Round to Transform Data Protection & Privacy via AI

    Funding includes OurCrowd.com, Prosegur (BME: PSG), and State of Mind Ventures

    Tel Aviv – Nov 13th, 2018 – Cognigo, the world’s first AI-driven enterprise platform for data protection and compliance, today announced the completion of a $8.5 million Series A round. The round was led by OurCrowd, with Prosegur (BME: PSG), and State of Mind Ventures. The new funding will be used to support Cognigo’s global sales and marketing expansion and product development, as well as further fuel investment in its Cognitive Computing technology, which helps organizations achieve data protection and privacy regulatory compliance (such as GDPR, PIPEDA, California Consumer Privacy Act and others).

    Cognigo CEO and Founder Guy Leibovitz said, “Data breaches are constantly expanding, and some have a disastrous impact on both the privacy of users and the security posture of the organization. In today’s era data has become the new currency. Regulations like California’s new privacy laws slated to come in effect in January 2020 and the GDPR, require organizations to up their data privacy and protection methodologies. Cognigo’s technology combines cognitive science, AI, and machine learning to reveal dark data at scale in real time and automatically enforce data privacy and protection regulations. We are excited to announce a strategic partnership with Prosegur which will help us expand our reach and deliver our unique solutions to an even wider audience.”

    “We are excited to be working with this impressive team of professionals,” said Alejandro Alonso, Managing Director of Prosegur’s Cybersecurity business. “Cognigo’s technology and solutions introduce a new and innovative approach for organizations to manage their data to comply with data privacy regulations such as the GDPR. We see the market’s demand for a practical and extremely efficient approach to GDPR. We are keen to kick-off this partnership with Guy and his talented team and together deliver innovative solutions that address a real and growing market challenge.”

     OurCrowd CEO Jon Medved said, “Data is today’s most valuable asset for both individuals and companies. Cognigo provides companies with the critical tools and systems they need to get control of their data. With today’s new privacy regulations such as the GDPR and the CCPA, control and protection of data Is crucial not only to avoid regulatory fines, but also to avoid embarrassment and business disruption.”

    About Cognigo
    Cognigo is the world’s first AI-driven platform for data protection and compliance, offering a single point of control to manage and secure critical data, sensitive assets and PIIs. Cognigo’s AI-driven solution provides a human-free data discovery, classification and protection platform, which supports structured and unstructured data – both for on-premise and cloud data silos.  For more information and to schedule a demo visit: https://www.cognigo.com

  • Former Google Sr. Software Engineer and Kythe Project Lead to Join Company Enabling Machine Learning for Code Analysis

    Michael Fromberger joins source{d} to lead Programming Language Analysis efforts

    San Francisco, November 7, 2018 — Today source{d}, the company enabling Machine Learning for large scale code analysis, announced that Michael Fromberger, former senior software engineer at Google, and technical lead/manager for the Kythe open source project (internally named Grok at Google), will join the company to lead its Language Analysis efforts.

    source{d} is now uniquely positioned to be the first independent company that brings the power of Kythe to developers in the open source community and enterprise. By combining the power of Kythe with the source{d} Engine, source{d} offers the most extensive Code As Data product in the market.

    “source{d} has developed some unique, powerful, and insightful open-source projects for language-agnostic source code analysis. After spending the past eight years designing and developing the Kythe project at Google, source{d} is a really compelling opportunity for me,” said Fromberger. “Historically, developers and engineering teams have had to reinvent a lot of the same wheels. As someone who’s passionate about both developer tools and open-source, I’m very excited to help address that issue. I believe the source{d} code-analysis and retrieval projects, combined with Kythe integration, will help boost developer productivity substantially.”

    Prior to joining source{d}, Michael spent eight years at Google working full-time on the Grok project, a high-quality graph-structured semantic index of source code. He was instrumental in re-designing and releasing Grok as an open-source project named Kythe.

    “Michael joining is a major milestone and vote of confidence for the company considering that Kythe is the data layer that powers all of Google’s static analysis over code,” said Eiso Kant, source{d} CEO. “With him on board, source{d} will be able to leverage Kythe’s capabilities to further help companies analyze and modernize their codebases, making it a more productive asset and facilitating the adoption of Inner Source practices.”

    About source{d}

    source{d}, the only open core company to turn code into actionable data and business intelligence, is building the tech stack that enables large-scale code analysis and machine learning on code. Used by top engineers at world leading companies, source{d} develops projects transparently, collaborating with the broader community of Machine Learning on Code researchers. Headquartered in Madrid, with US offices in San Francisco, source{d} has raised $10 million from Otium, Sunstone Capital and others. To learn more, visit sourced.tech.

     

  • Unlocking the Power of AIOps

    Unlocking the Power of AIOps

    AIOps is becoming quite the hot topic lately. It seems as though everyone is realizing that Ops is difficult. As we have to deal with more big data at greater velocity with more automation and complexity, there is no way humans alone can tame the beast. AIOps seems to be the answer.

    I recently sat down with Dave Link, CEO and founder of ScienceLogic to discuss how to unlock the power of AIOps to empower IT ops. We video taped the conversation, along with some accompanying slides that help explain what the challenges are that ops faces and what are some of the ways to tackle this using AIOps.

    I should mention that Dave and I are old friends, having worked together 20-plus years ago at a dotcom era company called Interliant. Since 2003 or so I have watched Dave build ScienceLogic into the powerhouse it is today. I couldn’t be more happy and proud of him.

    ScienceLogic has been named by EMA as a top vendor in the AI/ML for IT Operations. Dave gives us a good background on ScienceLogic as well as a good background on the use cases and challenges that IT Ops faces that AIOps seems to be the answer for.

    Dave also talks about the differences between AI and machine learning or ML. This seems to be an area of confusion today as well.

    The video is about 30 minutes and excuse Dave and I reminiscing, but there is some really great information here.

    Video on AIOps

     

    — Alan Shimel

  • DevOptimism: The Future of DevOps is Becoming Clearer

    DevOptimism: The Future of DevOps is Becoming Clearer

    DevOps has quickly matured from a fairy tale to a bestseller at most large enterprises. However, we are still at the beginning of the DevOps story, with many exciting chapters still to be written. This article looks at several emerging technologies shaping DevOps today and a handful of developing trends that will shape the ecosystem in the years to come.

    Today’s IT organizations increasingly are moving away from monolithic applications deployed through grand-slam releases with security slapped on after the fact. We are seeing many enterprises transitioning to more atomic applications released in continuous deployments with application security baked in from the beginning. Fueling today’s emerging DevOps is a bevy of new products focused on architectures including microservices, service mesh and function-as-a-service (FaaS); advancements in pipeline automation; and more adaptable security paradigms such as interactive application security testing (IAST) and runtime application self-protection (RASP). While these solutions herald a great step forward, they do demand that legacy applications be refactored into smaller granularity services, layered onto newer architectures and migrated to processes that allow greater automation.

    OK, that’s where we sit today: Great progress creating great opportunities and great challenges. But let’s head a little further down the road to understand where today’s advancements are driving DevOps tomorrow.

    There are five trends consistently appearing on the whiteboards of venture firms and forward-thinking startups that will shape DevOps in the upcoming years. At a macro-level, DevOps must become wider and smarter. As we dig layer deeper, we see five distinct patterns appear in which emerging technologies will make the release pipeline more insightful and proactive while allowing it to handle a wider spectrum of pipeline artifacts beyond just applications:

    • AI for DevOps: Artificial intelligence (AI) is making inroads into all facets of IT and DevOps is no exception. AI techniques soon will make the DevOps pipeline smarter, able to predict the impact and risk of deployments, spot procedural bottlenecks and identify automation shortcuts. Advancements in robotic process automation (RPA) lend themselves directly to optimizing the various handoff and automation points of the release pipeline. AI-based predictive analytics also will allow for higher fidelity operational capacity planning and pre-deployment fault prediction.
    • DevOps for AI: Today we are seeing a handful of new tools, commonly called MLOps, emerging on the market that allow developers and modelers to deploy new AI models rapidly and repeatably. However, these products have yet to collaborate closely with the Ops side of the enterprise. They are generally a development aid, not an operational aid—or said another way, they are all “ML” and very little “Ops.” This space will mature rapidly, however, allowing release of AI models via the DevOps ecosystem so as to include operational aspects. Conversely, tomorrow’s DevOps pipeline will also allow AI models that have been learning from real-world data to be extracted from production and moved back into the development environment.
    • DBOps: Today, new databases, schemas, indexes, triggers and stored procedures typically are deployed manually by DBAs outside the DevOps pipeline, with their operational impact being provided by best-guess capacity planning. This glaring gap soon will be closed by products that allow databases and their associated artifacts to be released through the deployment pipeline along with the applications with which they are coupled. This will not just encompass traditional relational databases, of course, but also support non-structured databases and object stores. The prime objective being that applications and their supporting data repositories will be developed, tested and deployed via the same automated pipeline as applications without the need for specialized database handholding.
    • DataOps: When new apps are deployed, the associated analytics and visualizations are typically brought online by data scientists after the fact through manual deployments. Looking forward, expect data analytics to be introduced into the DevOps pipeline soon after the requirements for the release are defined. Much like DBOps above, the goal is to eliminate specialized manual promotion. But beyond just synchronous deployment, DataOps will allow data governance, data loss prevention (DLP), data lifecycle management and copy data management features to be integrated during each new deployment.
    • Blockchain for DevOps: Today’s and tomorrow’s DevOps evolutions will increase the velocity, variety and volume of pipeline traffic dramatically. While this sounds promising, it undoubtedly will be a nightmare to audit, verify compliance and ultimately control and manage. One technology being eyed to untangle this spaghetti is a distributed consensus mechanism such as blockchain. Stop rolling your eyes—we’re not talking about Bitcoin or ICOs. We are describing the need for a centralized, permissioned, secure, distributed ledger to record, track, authenticate and audit the torrent of pipeline transactions. Blockchain seems to fit the bill nicely and also has the added benefit of enabling digital contracts at each pipeline stage that can be verified at runtime to trigger artifact promotion and also audited long after the fact to ensure that all rules were followed.

    DevOps has already become the standard deployment paradigm in many enterprise IT shops and its adoption will continue to become more pervasive. But this growth requires DevOps products and processes to become smarter, more secure and more controllable as the release pipeline incorporates a wider set of deployment artifacts beyond just applications. This is a daunting challenge to be sure, but there are exciting new products that will emerge to help write the next chapters in the DevOps story.

    — Mark Campbell