devopsy.kinsta.cloud

Category: AI

IPsoft’s 1Desk Advances AI Labor Force Reality

IPsoft wants to put a digital face on the future of IT operations that will be infused by artificial intelligence (AI). Its recently launched 1Desk service desk combines the Amelia digital assistant with IPsoft’s Apollo virtual engineer technology to automate multiple backend processes.

Allan Anderson, director of enterprise solutions for IPsoft, says 1Desk effectively combines two longstanding areas of IPsoft research to provide IT organization with an ability to manage IT at scale using digital personal assistants and virtual engineers.

IPsoft has been providing managed services based on its Apollo AI framework for several years. The launch of 1Desk creates an opportunity for IT organizations to not only employ that AI framework on their own, but also combine it with a digital assistant technology christened Amelia that IPsoft has trained to handle a variety of technology and business process requests.

Anderson says the virtual engineer AI technology developed by IPsoft make use of machine learning algorithms to continually monitor how humans complete a task. Armed with that data, those virtual engineers can then incorporate that information into their knowledge base to learn a process or fine-tune an existing one.

A continuing shortage of IT skills in key areas is significantly reducing resistance to AI technologies being applied inside IT organizations—AI technologies may enable organizations of almost any size to operate IT environments at unprecedented levels of scale while minimizing human errors. IPsoft, says Anderson, expects virtual engineers to be employed as a digital labor force capable of performing tasks such as identifying and killing a long-running query to configuring and executing runbooks across the most complicated data center environments. To facilitate the automation of the those processes, IPsoft has developed a Digital Labor Studio tool that organizations can use to create and train a virtual agent.

There’s no doubt that many of the tasks associated with managing an IT environment are repetitive. In fact, any repetitive IT task is arguably a ripe candidate for automation. How those advances will impact DevOps processes remains to be seen. It’s not difficult to imagine that each development team will be assigned its own digital assistant. The role of the IT operations staff then would evolve into one where a smaller group of people train and manage a pool of digital labor.

IT professionals are generally used to the idea that IT will be used to automate a task that used to be part of someone else’s job. But now IT is being used to automate many of the tasks performed by IT professionals. It may take some time for them to first accept that idea and then evolve their skills accordingly. There will also be a need for some level of human involvement when it comes to IT. But instead of thinking in terms of managers and administrators, perhaps the brave new world will be one where everyone in IT is now the manager of their own digital force of labor.

— Mike Vizard

November 17, 2017
AT&T Democratizes Acumos Code for AI

AT&T and India IT outsourcing company Tech Mahindra announced that code the companies jointly developed to drive artificial intelligence (AI) applications is being contributed to the Linux Foundation.

Mazin Gilbert, vice president of advanced technology for AT&T Labs, says the goal of Acumos is to not only democratize access to AI software, but also consume it as a microservice with a standard set of DevOps processes.

Gilbert says AT&T decided to contribute Acumos to the Linux Foundation because it’s become clear the proprietary implementations of AI platforms will prove to be too difficult and costly to support. AT&T is addressing this issue because it currently has 10 proprietary AI platforms attached to its network and was looking at having hundreds attached within a matter of months.

Making Acumos available as an open-source project should go a long way to reducing that complexity by providing a standard interface to integrating various AI platforms, says Gilbert. Acumos includes a standard mechanism for downloading AI code from an app store, bringing AI functionality to where data resides instead of requiring developers to load data into a proprietary cloud service to access AI functions. In the case of AT&T, that integration is accomplished via the company’s Indigo software-defined network (SDN).

Gilbert notes that much of what passes for advanced AI today has been around for decades. Many of the algorithms being used to drive AI applications are now over 50 years old. The only real differentiation is how these algorithms are being combined to drive applications to access enough data to make the algorithms useful. Because of that simple fact, AT&T doesn’t believe giving Acumos to the Linux Foundation represents any major loss of intellectual property.

As AI becomes more democratized, DevOps teams will manage massive amounts of distributed data pipelines. AI applications require access to massive amounts of data that must be consumed via a consistently applied information architecture. Sometimes referred to as DataOps, the pipelines created by operations teams enable the algorithms embedded in a developer’s application to continually learn about changes that impact the recommendations being surfaced by the AI engine.

At this point it’s now only a matter of time before every application is infused with some form of AI. In many cases, the front end to those AI capabilities will be a voice-enabled digital assistant trained to optimize a specific processor or answer a broad range of capabilities. Applications that don’t provide these capabilities soon will be deemed archaic. The result from a DevOps perspective should be a massive series of rolling upgrades to infuse AI capabilities into existing applications or replace them with one that provide those capabilities. Regardless of how that is accomplished, without some form of integrated DevOps processes in place there can be no data for AI applications to consume.

— Mike Vizard

November 8, 2017
Measure What’s Important: DevFinOps

Lately there seem to have been quite a few DevSomethingOps terms being coined. However, I do think we need one more: DevFinOps.

Dev-What-Ops?

One of the biggest problems in IT is working out the cost of IT. Left unchecked, IT will expand to fill whatever budget it is given. This may mean evermore involved business continuity and redundancy, reinventing evermore elaborate wheels, irrational versus buy decisions and so on.

In fact, the whole idea of cost estimates for IT projects has become something of a joke. Most IT projects are reputed to exceed their budgets.

There is an entire #NoEstimates movement that rejects the idea that it is even possible to deliver a sensible estimate for a software development project. As is often the case, the basic idea is reasonable:

it is possible to do small chunks of work incrementally, leading as rapidly as possible to a desired shippable product, and that when you do that there is no need to do much of anything in the way of estimating stories or the project.

However, the simplistic interpretation that is all over Twitter is that because there is no estimate of the overall project, there is no estimate at all. That is not the idea! It is simply to break down the overall estimate (which is probably wrong) and instead budget for much smaller chunks of work, which can presumably be estimated more precisely and accurately.

So What Can We Do?

Maybe what is required is integration of financial concerns directly into the fabric of IT development operation. We might call this DevFinOps, in line with other extensions of DevOps such as DevSecOps or DevNetOps.

This is not such a leap; after all, we already integrate costs in areas such as SLA calculation, estimating the TCO of a tool or the ROI of a proposed purchase. However, these are generally calculations that are made after the fact, and may even be retconned to fit external assumptions.

What I am proposing, though, is to build this information right into the IT metadata and processes to enable us to make better decisions.

This is not IT Asset Management, although it would build on those data and extend their reach. A particular device or piece of hardware has its purchase price and current amortised value documented in the ITAM database, but the actual value it has at any given time might vary depending on which business function it is currently supporting. This calculation should change over time to reflect the real current value it is delivering to the organization at any given time.

An idle server has no particular value beyond the intrinsic value of the hardware, back-end support (floor space, network, cooling, etc.), and perhaps any associated licensed software. However, the same server will take on a much higher value to the organization when it is provisioned as part of the infrastructure supporting a major business-critical application.

Performing actions on these systems would therefore have different potential impacts, and should be treated as costs deducted from a limited budget. Live-patching a business-critical server is an expensive operation, not to be treated lightly. The cost of doing so should be deducted from an operational budget.

This calculation has the benefit of making it easier to communicate the different impacts of various proposed courses of action outside the IT department. Deciding between cloud and on-premises deployment options, or between different flavors of cloud, can be a fraught process. Technical options and pricing models are rarely easily comparable, and then there are other costs such as compliance (yes, and GDPR) or vendor risk. Quantifying and normalizing these disparate factors would make it that much easier to compare options up front and calculate benefits after the fact.

Is This Even Possible?

Google already does something like this in its Site Reliability Engineering (SRE) program, with the notion of an error budget:

The business or the product must establish what the availability target is for the system. Once you’ve done that, one minus the availability target is what we call the error budget; if it’s 99.99% available, that means that it’s 0.01% unavailable. Now we are allowed to have .01% unavailability and this is a budget. We can spend it on anything we want, as long as we don’t overspend it.

There is a long and sordid history of attempts to make programming projects comparable and their estimates more reliable. Often this devolves to simply counting lines of code, according to the age-old management syllogism: “We need to measure something; this measurement is easy to gather; let’s measure this!” and of course the Mythical Man-Month.

Fortunately, there is a way to calculate the complexity, and therefore the potential cost impact, of a piece of code: the Halstead volume. This is a measurement of the complexity of code, described in terms of the number of distinct operators, the number of distinct operands and the total number of operators and operands. While it is not guaranteed to be perfectly accurate in every case, it is at least accurate enough to be comparable, and therefore can be used to estimate the risk of a change, in terms of the relative Halstead volumes of the code to be changed and the patch to be applied.

But What About the Squishy Meatbags?

Connecting the human costs would also enable other metrics to be gathered. It has been historically difficult to differentiate between what is urgent and what is important, and how to prioritize each. If we connect salary data (including overtime) to changes, it becomes much easier to make the call on when enough time has been wasted on a problem and the cost-effective solution is to start from scratch.

It sounds like a lot of work to gather all of this information, but these data already exist; they’re just not connected up in one place. When the business wants to launch a new service, there are (usually) pretty rigorous calculations behind that decision. The expectation is to acquire or retain a certain number of customers who have a certain average lifetime value and break-even points and other criteria are all mapped out ahead of time. For whatever reason, though, these islands of information are rarely if ever connected up.

This, incidentally, is just one reason why it can be worthwhile to work with reputable and experienced vendors. I have written before about what this process looks like. The good vendor sales teams will help to unearth this sort of information—but even in the most positive sales processes, the kimono is rarely fully open. There is far more information inside the organization that could be enormously useful if only it were available in the right place and at the right time.

Longer term, this would mean that business cases and ROI calculations would finally be concrete and factual, rather than extended performance pieces in the medium of spreadsheets. On that note, there is a wonderful little book, titled “How to Lie with Statistics,” which is even available as a free PDF. It’s one of those books that is fun to read because you start spotting the things it was talking about simply everywhere.

If you wanted to be maximally Zeitgeist-compliant, you could probably verify all of this with blockchain, although I worry that the transaction costs (including thermal costs!) would be prohibitive. The good news, though, is that you could use this DevFinOps methodology to work that out too …

All That Effort, for What?

Bottom line, the goal is to be able to calculate (at least within reasonable limits) how much IT is costing today, and what the impact of any proposed changes would be—and crucially, to do so in real time.

The goal is agile cost estimates. There is a reason I chose the DevFinOps coinage. The way we do cost estimates today is very “waterfall“: large teams have extended arguments over spreadsheets, and the process is so painful and expensive (in person-hours and disruption) that it is only undertaken for the largest changes.

Agile’s faster release cycles do not align at all well to this process, which is where #NoEstimates is coming from. However, if we start gathering and processing all the cost data in real time, and make it available via a simple API call, it becomes possible to calculate both ends of the cost of a proposed change: the complexity of the code itself, and therefore the probability that it contains bugs, and the infrastructure impact that the deployment will have. This cost can then be compared to the team’s current SRE error budget to determine whether the change can be made automatically or whether it requires signoff as an exception.

The result looks a lot like ITIL’s notion of a “standard change”, but without the inherent conflict between ITIL’s centralization and SRE’s decentralization. The key is a common accepted measurement of what a standard change actually is, which would otherwise require the ITIL Change Manager to make the determination ahead of time.

What Does This Look Like in Ops?

Let’s not forget Ops, either. Proper cost estimates like the ones described above would enable Ops to prioritize alerts correctly and estimate impacts accurately. Two outages with equal technical severity may have very different business impacts. Today, that reconciliation relies on tribal knowledge and systems that are often inaccurate or out of date. Given a central shared repository of data that describes the value of deployed systems, it becomes possible to calculate those impacts much more accurately.

While we work to get to full DevFinOps, you might want to consider AIOps, which helps gather a contextual view of what is actually going on in IT. Gartner just released a Market Survey for this emerging space. Normally these documents cost $1,295, but you can get a free copy here, courtesy of Moogsoft.

Dominic Wellington is Chief Evangelist at Moogsoft. He has been involved in IT operations for a number of years, working in fields as diverse as SecOps, cloud computing, and data center automation.

— Dominic Wellington

October 27, 2017
The Need for Speed Demands New-Era APM
Applications have become the primary conduit by which customers experience a brand, making the application experience critical to business success. As such, continuous delivery and highly available systems that deliver optimal end user experiences at lightning speed have become top priorities for organizations. Modern IT is all about speed, efficiency and customer experience. Simply put, companies that develop new applications and services the fastest will win.

That’s where containers and microservices come in. Containerized microservice applications enable organizations of all sizes to approach the ultimate goal of continuous integration and continuous delivery. This accelerates everything within the application life cycle, from initial development to component updates and new functionality.

To match the development speed that microservices and containers enable, Ops teams need tools that can also operate hyper-fast, providing an immediate understanding of the impact of changes made, real-time analysis of vast amounts of data across applications and technologies, visibility across a range of languages and infrastructures, and knowledge of how the application is performing for end users.

In fact, we are at a critical inflection point in application performance management (APM), where a generational shift is needed to keep pace with the new hyper-dynamic application architectures consisting of microservices, containers and hybrid cloud environments. To fully understand the impact of this shift, we can examine the genealogy of APM.

APM Evolution

The first generation of APM, established at the end of the last century, collected statistics and metrics out of system and application components, providing a first generation of user-defined transaction profiling by using instrumentation technology. APM 1.0 required a lot of effort to implement and maintain, but helped to monitor and analyze the performance of newly developed 3-tier applications.

The second generation of APM was driven by more distributed, service-oriented architectures (SOA) and the rise of more elastic, virtualized or cloud-based infrastructure. APM 2.0 added application topology discovery and could analyze distributed applications. User experience monitoring of web and mobile apps quickly became part of APM 2.0 as more businesses depended on the service quality of their applications. With a primary focus on application monitoring, not management, APM 2.0 required an even broader set of subject matter experts to configure and interpret monitoring data from dashboards and reports. Similar to APM 1.0, the second generation still required way too much human analysis and setup to fully understand application problems and derive actions.

The continuous change in today’s modern dynamic applications makes it difficult, if not impossible, for those older tools to keep up. Throw in additional architectural challenges such as hybrid clouds, containers and orchestration (such as Kubernetes and Mesosphere) and the picture gets murkier, all in the name of high-speed delivery. Meanwhile, the microservice architecture and development strategy spawns wide technology diversity—each engineer can choose the programing language, middleware and databases best suited to code their individual microservice. Scale, complexity and constant change is the new reality for today’s dynamic applications.

Wanted: A New Breed of APM

In this environment, traditional IT monitoring tools won’t work. These new applications use a vastly larger set of technologies, must scale dynamically and are in a constant state of flux.

In this effort to engineer for speed and efficiency, complexity and scalability have increased exponentially, requiring continuous monitoring and more visibility than ever before. As organizations embrace agile methodologies for development and containerized microservices for deployment, developers have shifted their thinking in technology choices. Developers now continually select the most optimal technology for their individual goals regardless of DevOps’s ability to configure, deploy and manage the specific technology.

With several teams across the enterprise constantly publishing new software releases, understanding the impact of these new releases and changes to the overall environment has become a significant challenge. Today there is no single source of truth assisting communication between various organizations, resulting in what seems a wild west–like environment for IT. With so many groups and technologies in play, everybody seems to be speaking a different language, especially when it comes to delivering high-quality applications. In this environment, a new breed of APM is needed that can provide a “Truth Machine”—complete with deep visibility across all application components in the technology stack.

This new breed of APM must be automatic and dynamic to deal with the ever-changing world of applications running in containerized microservices. To deliver these critical capabilities, the tool must provide real-time visibility into dynamic, containerized, microservice applications, regardless of language or infrastructure. New thinking is required to meet the challenges of these complex, large and dynamic application environments; a new monitoring and management approach that can handle dynamism by:
- Delivering real-time analysis of vast amounts of data across all applications and technologies.
- Conducting continuous discovery and automatically mapping every component, host and container, (hosted on-prem, private cloud, public cloud), to the exact architecture of complete technology stack.
- Visualizing the application architecture and delivering actionable suggestions that assure the operational health of systems and applications.
- Providing real-time accuracy to visualize, structure and identify problems immediately.
- Monitoring the complete set of diverse technologies that run in microservice environments.
- Operating in a heterogeneous environment and automating deployment across nearly all programming languages.
Artificial Intelligence for Next-Generation APM

DevOps and IT organizations are struggling to match the speed of agile development, and this is creating exponentially more challenges around new application architectures. The only constant in these modern application environments is change. Whether change is due to microservice architectures, container orchestration or simply more agile delivery execution to effectively manage microservice applications, DevOps needs up-to-date visibility, a precise picture of the application’s structure and an immediate understanding of its health. To accomplish that, DevOps must be able to apply artificial intelligence to the act of monitoring modern dynamic applications.

Rather than spending time configuring and reconfiguring monitoring tools, organizations must automate management steps with as little human setup and configuration as possible. To have complete control and visibility of their environments, regardless of language and infrastructure, organizations must use a modern application management solution that is automatic and AI-powered to deal with the ever-changing world of dynamic containerized microservice applications.

About the Author / Pete Abrams

Pete Abrams is a co-founder and the chief operating officer at Instana, the leader in application performance management for containerized microservices applications. Formerly vice president of customer centric innovation at AppDynamics, Pete has spent more than two decades in the technology industry focused on introducing new products to market and ensuring customer feedback and desired capabilities are incorporated into future product innovation. Connect with him on LinkedIn.
October 27, 2017
Research Firms Agree, Cloud Growth Continues to Soar

Investment in all the various types of cloud computing has gone gangbusters in recent years, and for good reason: Cloud computing helps to cut cost, shorting time to market, provides unrivaled scale and reliability.

This will only continue as enterprises continue to divest from on-premises infrastructure over the long term while seeking to scale the tremendous data generated by AI, IoT, machine learning and digital transformation. It looks as though for the few years ahead, it’s nothing but tailwinds for cloud investment and growth.

According to research firm Gartner Inc., the software-as-a-service (SaaS) market reached $48.2 billion in 2016, growing more rapidly than the firm previously projected—although Gartner didn’t say in the release what its previous revenue prediction had been.

Currently, Gartner expects the total worldwide public cloud services market to grow 18.5 percent in 2017 to reach $260.2 billion. That’s up considerably from $219.6 billion in 2016.

According to Gartner, SaaS revenue is expected to grow 21 percent this year to reach $58.6 billion. “The acceleration in SaaS adoption can be explained by providers delivering nearly all application functional extensions and add-ons as a service. This appeals to users because SaaS solutions are engineered to be more purpose-built and are delivering better business outcomes than traditional software is,” Gartner writes.

The highest-revenue growth will come from cloud system infrastructure-as-a-service, which Gartner says is projected to grow 36.6 percent in 2017 and hit $34.7 billion.

Cloud Mojo Keeps on Rising

Cloud is only going to continue to take a greater share of overall spend in the years ahead. According to Sid Nag, research director at Gartner, last year about 17 percent of the total market for infrastructure, middleware, application and business process services shifted to cloud, and by 2021, that will rise to 28 percent.

Those looking at trends outside of cloud won’t be surprised to see cloud growing so much greater than that of the overall global economy. According to research firm IDC, worldwide information and communications technology (ICT) is set to rise over the next five years, fueled by IoT, robotics, augmented and virtual reality, cognitive computing and artificial intelligence.

Total ICT spending on traditional IT, telecom services and new technologies will grow to $5.6 trillion by 2021 from $4.3 trillion in 2016, representing a compound annual growth rate of 6 percent in constant currency terms. Annual growth will accelerate through the forecast period, to 6.5 percent in 2021 from 5.5 percent in 2017, as new technologies account for a growing proportion of the overall market, IDC writes.

Of course, all of this will increase investment in security and, according to a recent report from Markets and Markets, the cloud security marketing, including identity, data leak prevention, intrusion detection/prevention systems, encryption and security information and event management systems. According to Markets and Markets, the cloud security market will grow to $12.73 billion by 2022, up from only $4.09 billion in 2017.

— George V. Hulme

October 24, 2017
BMC to Infuse AI Across Software Portfolio
At its recent BMC Exchange event, BMC Software announced it will be infusing machine learning algorithms and other artificial intelligence (AI) technologies across its entire portfolio of IT operations software.

The first instance of BMC software to make use of machine learning algorithms is the latest version of TrueSight AIOps platform, which ingests data from multiple data sources to automate processes across BMC service desk, IT operations and tools for managing DevOps processes. BMC also is working with IBM to employ the cognitive computing services known as the IBM Watson cloud service across its portfolio.

In addition, the company launched TrueSight Cloud Cost Control, a tool for managing and tracking usage of cloud services across multiple providers, and BMC SecOps Policy Service, which easier to embed compliance and testing via an external service that gets included in their software development processes.

Finally, BMC announced it has signed an alliance with Amazon Web Services (AWS) under which the two companies will work to accelerate the migration of application workloads to the AWS public cloud.

Dan Streetman, a senior vice president for worldwide sales and marketing at BMC, says his company is developing a product strategy around three major themes:
- IT organizations will require management tools that will work across multiple clouds;
- As part of general “shift left” developers now expect to be able to programmatically invoke IT operations and security services; and
- IT professionals now expect the tools they use to manage IT to have the same types of advanced capabilities that consumer applications provide via speech interfaces such as Apple Siri or Microsoft Cortana. Various IT and customer service desks also soon will use speech interfaces to provide a highly automated self-service experience to end users.
In fact, given the complexity of the IT tasks that now need to be managed, Streetman says a race is now on between IT vendors to live up to those expectations. Much less of a concern, he says, is any notion that AI technologies might one day replace IT professionals that by and large are already overtaxed; rather, it is apparent that IT vendors are now being evaluated on their ability to maximize existing investments in IT professionals that are already in short supply. The real opportunity is to free IT professionals to enable them to concentrate on higher-value tasks.

Naturally, it remains to be seen how various AI technologies will impact the management of IT. It certainly will take a while for IT administrators to put their faith in AI. But as those technologies continue to mature, there may soon come a day when IT professionals look back with wonder at all the things they used to have to accomplish one manual task at a time.

— Mike Vizard
October 19, 2017
Machine Learning Is Working for CIOs, Vendor Survey Finds
AI’s time has finally come, if a new survey commissioned by ServiceNow is to be believed. Its results showed that nearly 90 percent of CIOs are at least dabbling with machine learning, with 53 percent calling it an area of strategic focus for their companies. What’s more, almost 70 percent of the CIOs surveyed say decisions made by machines will soon be more accurate than those made by humans.

The new survey was conducted by Oxford Economics on behalf of IT and enterprise software-as-a-service vendor ServiceNow. The survey, titled “The Global CIO Point of View,” was completed by 500 CIOs in 11 countries across 25 industries.

The report also fits neatly with evidence from other sources that CIOs are in charge of digital transformation. Just under 75 percent say they lead their company’s efforts to gear up for full-fledged digital business. Business intelligence derived from data analytics has always been a pillar of digital transformation. But with some 85 percent of CIOs reporting that machine learning provided “substantial value” or “transformative value” to the accuracy as well as rapidity of decisions, it’s beginning to change the way some organizations operate, according to the report.

Within three years, two-thirds of the respondents expect to be making an investment in machine learning. Today, one-third are investing. The size of spend as a percentage of the overall IT budget is relatively modest, however: something like 5 percent for machine learning.

One of the reasons for the move to machine learning is that sometimes you just have to do things smarter. “We were heading to a tipping point almost where there was just so much work coming in,” said Dave Wright, chief innovation officer, ServiceNow, talking about IT shops. “There was just so much work coming in; the workload just continued to increase year over year. CIOs want to know, ‘How do I get to the point where I can deal with this, because I can’t constantly throw heads at it.’” That’s why the report advises that companies direct their early machine learning efforts at the most commonly used services, freeing resources for more rewarding efforts.

Machine Learning’s Obstacles Aren’t What You Think

Machine learning technology has matured so much in recent years that it’s no longer the prime obstacle for CIOs. Instead the challenges are outdated business processes, incomplete insufficient skill sets, lack of quality data, insufficient data sources, risk assessment conflicts and budget limitations. When data is siloed across the company, accessing all the right data for the best decision-making is not always easy or possible. There’s also some inertia built into the organizational structures of most companies; the notion of making business decisions based on machine learning (a.k.a. “decision automation”) may require some handholding, for example. Technology is not the problem.

Early adopters to machine learning, which represent just under 10 percent of the survey sample, share some characteristics and ways of doing things: They prioritize talent. They are working on better business processes and have a road map for future changes developed by monitoring mistakes made by machines. The road map is also guided by the need to implement policies and processes that improve data quality. They have strong technology foundations, with an emphasis on data analytics, cloud, mobile and IoT. And they expect results. Nearly 90 percent of this 10 percent of early adopters expect decision automation to support topline growth.

What may be more interesting is that, of the other 90 percent of the CIO respondents, nearly 70 percent expect to develop revenue from machine learning. Looking at the data behind this report, the CIOs who’ve gotten their feet wet with machine learning are very excited by its prospects.

The report outlines five steps to achieve value from machine learning that are worth a quick summary:
1. Build your technical foundation so as to improve data quality. Successful companies had already invested significantly in analytics, for example.
2. Prioritize automation of the most common business use cases; they will return the greatest business benefit.
3. Build an exceptional customer experience. If customers see the man behind the curtain in a machine learning-based tool, it may become a very negative experience for them. Spend extra time making the entire customer journey seamless.
4. Attract employees with new skills and double down on culture.
5. Measure and report. The benefits of machine learning may not be obvious to all. CIOs must set expectations, develop success metrics and build a sound business case to acquire funding. Build some success stories.
— Scot Finnie
October 17, 2017
Using AI to Automate Incident Management Sparks Debate

When it comes to incident response with an integrated DevOps environment so much depends on the ability of an IT team to respond quickly and consistently to a specific set of events. Because of that requirement, many IT organizations have invested in incident management systems that everyone in IT can consult whenever an issue arises. Now PagerDuty, a provider of incident management software, is taking that concept a step further by applying artificial intelligence (AI) to automate as many incident response tasks as an organization deems appropriate.

As promising as that may sound, however, Resilient Systems, a unit of IBM that is an arch-rival to PagerDuty, contends most organizations today do not have anywhere near the level of incident management maturity to make investing in AI worth their time and effort yet.

Eric Sigler, head of DevOps for PagerDuty, says the the latest edition of the company’s namesake incident management platform is designed to augment the capabilities of the local IT administrator by providing:

Alert Grouping: Rules-based automation and machine learning algorithms automatically group related issues together to make it simpler to identify related issues.

Similar Incidents: IT administrators and support specialist can see previous related issues and surface information around incident severity, impact and remediation steps.

Response Automation: A Response Plays capability allows teams to design and execute an automated response pattern to recruiting responders and stakeholders by a single click.

Dynamic Notifications and Event Routing: IT teams can select notification and assignment behavior and automatically route events to different teams based on event payloads.

Sigler says PagerDuty has also redesigned the company’s user interface, creating an enhanced live incident details page that makes discovering events simpler.

Those capabilities, says Sigler, will collectively prove to be critical in an emergency because most of the time the specialist associated with a specific IT discipline is usually not immediately available when needed. First responders to any given event need as much information as possible as their fingertips to resolve issues as quickly as possible.

But Ted Julien, vice president of product management for Resilient Systems, says most organizations simply don’t have incident response processes in place that would benefit from that level of AI. Machine learning algorithms require access to massive amounts of incident response data to learn what to do, but most organizations don’t collect that data, much less learn from it, he says. That doesn’t mean AI doesn’t have potential. But he believes AI is not nearly as far along when it comes to incident response as anyone would like. In fact, he notes, machine learning algorithms and other forms of IT automation could just as easily wind up doing more harm than good.

There may come a day when incident response in an IT environment is fully automated. But the distance between where most IT organizations are today and the achievement of that goal is still nothing less than considerable.

— Mike Vizard

October 2, 2017
AIOps: Managing the Second Law of IT Ops

Or, why IT systems keep breaking

The tension between the two halves of DevOps is long-standing, going back long before the term was even invented. What underlies the split between Development and Operations is a fundamental difference, like an actual physical law. Let’s explore that analogy together.

IT Ops is fundamentally a constant fight against entropy, meaning disorder. Sysadmins expend significant energy trying to keep everything up and running, in a process which can sometimes feel like trying to shore up a sandcastle while the tide is coming in. Systems are constantly breaking down; users need changes made that introduce risk; and, as if that wasn’t enough, the whole shebang is constantly under attack from malicious outsiders. It’s no wonder that sysadmins have a reputation for being a bit grumpy!

It’s not just a good idea, it’s the law.

In contrast, Dev actually increases the level of entropy. Every new release, be it as small as a one-line fix or finding that errant semicolon, is a change that has to be made, reducing the order that Ops fights so hard to maintain. Dev is entropogenic, embracing change as a positive force.

The interaction between Dev and Ops is analogous to the second law of thermodynamics: Entropy can only increase over time, unless energy is introduced to the system from outside. In other words, unmanaged change will eventually cause something to break, unless people put in continuous effort to prevent those breaks.

Thermodynamics: Lots of Heat, not Much Light

These conflicting goals, taken to extremes, can get unhealthy, with Ops actively fighting change or Dev going around Ops to increase their velocity of change. Left unchecked, this tension can turn into a vicious cycle, as frustration with the “Department of No” drives developers into unmanaged “shadow IT.”

The good news is that the analogy with hard laws of physics is not perfect. The process of increasing entropy is not irreversible, and does not require constant inputs of energy to prevent. New agile approaches to IT Ops are emerging that are light and flexible, taking their structure from that tension. In the physical world, this is called “tensegrity,” and was originally formulated by Buckminster Fuller, one of those annoying polymaths who seem to make breakthroughs in every field they touch.

Tensegrity structure

IT Ops needs its own equivalent to tensegrity, a flexible, holistic structure that is able to embrace change while maintaining its shape. Structures that try to resist change through rigidity are brittle and ultimately fragile.

Today, this flexible strength is enabled by AIOps, a term coined by Gartner to describe a new way of operating IT, based on augmenting human skills with automated analysis enabled by modern algorithmic and machine-learning approaches.

Complexity Breeds Complexity

Modern IT is too complex for any single person or even team to keep track of, instead requiring multiple specializations. In turn, each of these functions requires its own specific support tools that enable that particular role. Beyond a certain point, it becomes very difficult to combine these partial views to gain a holistic understanding of what is actually happening in the environment.

Because of the scale of modern IT environments and the sheer rate of change that businesses require, old-style rules-based “Managers of Managers” can no longer keep up, requiring more effort to build and maintain their worth. Especially as infrastructure becomes more and more self-modifying in response to changing conditions and demand, IT Operations needs to accelerate just to keep up.

AIOps brings developing issues to the attention of sysadmins before they become visible to customers and end users, and gives them the tools to collaborate effectively between different teams and specialist roles to avoid outages or restore service quickly and efficiently. In this world, automation is used to augment the capabilities of the human operators to make them more effective, letting people concentrate on adding value, not just fighting fires.

This is how Ops can get out of the business of fighting Dev, and instead become a supportive partner. AIOps can catch issues that are affecting services and flag them to the right people early on, including the DevOps engineers that are responsible for production support.

For developers, one of the fears around DevOps is that in “picking up the pager,” they are opening themselves up to a constant stream of irrelevant alerts. Old Ops hands nod wryly at this, or maybe try to share their personal approach to filtering the noise. Instead, AIOps ensures that only valid, actionable alerts get as far as being shown to a human being and taking up their valuable time.

Eyes on the Prize

It’s not all roses. As the AIOps market begins to mature, there is a certain amount of “algorithm-washing” going on, with more and more vendors trying to claim that either their legacy tool was AIOps all along, or that their one-trick pony is actually a whole three-ring circus. What gets lost in the excitement is what all of those algorithms are actually supposed to achieve.

Me too!

In considering a shift toward AIOps, the goal should not be simply to build a better mouse trap, meaning a smarter filter for monitoring events. The success of the project can be measured not so much in the mathematical results of the algorithms, but in the smoother exchange between the human users. This easier collaboration is especially important across team boundaries, whether these be different specialist units within the Ops function or developers and software engineers who need to work with Ops to understand the performance of their code in production.

The benefit of transforming operations this way—moving from sequential, waterfall Ops to abile, collaborative Ops—is to turn DevOps-enabled teams into a sort of perpetual-motion machine, able to continue operating without requiring constant massive inputs of energy from outside the system. Modern applications and the complex infrastructures that support them cannot be managed with the sorts of rigid, cobbled-together, industrial approaches that were appropriate in the mainframe days. AIOps is the agile, flexible approach that can not only accomodate that complexity, but thrive on it.

— Dominic Wellington

September 22, 2017
New Relic Delivers on AI Promise to Automate IT Operations

New Relic at a FutureStack 2017 conference unveiled an instance of its monitoring tools that employs machine learning technologies and other forms of artificial intelligence (AI) to provide advice on how to optimize an IT environment.

Nadya Duke Boone, director of product management for New Relic, says New Relic Applied Intelligence (NRAI) will be made available as part of the company’s Digital Intelligence Platform. In addition, New Relic is making available three dedicated NRAI capabilities in the form of Radar, NRQL Baseline Alerting and New Relic APM Error Profiles services. These services are designed to help IT operations teams spot abnormal behavior across billions of pieces of data and predict where problems may arise. The services ultimately will surface prescriptive recommendations for problems before they occur.

The goal, says Duke Boone, is to leverage New Relic’s software-as-a-service (SaaS) approach to IT monitoring tools democratize access to AI technologies. That’s possible, she says, because of the massive amount of data New Relic collects. That data is then fed into machine learning algorithms to detect patterns indicative of a variety of potential issues.

Duke Boone says that intelligence is then delivered as a personalized Radar news feed that is similar to a social media news feed. That news alert might, for example, might inform an IT administrator that, based on characteristics of their IT environment, an issue is likely to arise. IT operations teams also have the option to integrate that Radar news feed with a collaboration platform such as Slack to alert the rest of the IT operations team to a potential issue.

NRQL Baseline Alerting allows IT operations team to set thresholds for specific applications that when exceeded will generate an alert. New Relic APM Error Profiles allows IT administrators to launch queries against NRAI to detect potential issues that might arise from outlier applications in the IT environment, adds Duke Boone.

New Relic doesn’t see AI replacing IT operations teams as much as augmenting them, Duke Boone notes. Most organizations can’t hire enough IT staff to manage their IT environment. As those IT environments become more complex, it’s becoming clear that IT professionals will need to leverage AI technologies to keep all the applications they deployed running optimally, she says.

New Relic has been investing extensively in AI ever since it previewed these capabilities in the form of a Project Seymour initiative it revealed last year. Now the company and its rivals are locked in an arms race to develop and implement machine learning algorithms that should enable IT operations teams to manage applications at unprecedented levels of scale.

In some quarters AI is viewed as an existential threat to the IT operations teams. In reality, AI will be used to automatically resolve low-level issues that collectively conspire to rob IT organizations of the time they need to focus on new innovative ideas that add value to the business. In fact, assuming many of those issues will be soon disappear it’s conceivable that all the drudgery associated with managing IT today made soon just fade away.

— Mike Vizard

September 14, 2017