VMware is moving to make it easier for organizations to automate data centers and public cloud based on its core hypervisor technology by injecting advanced analytics into its VMware vRealize Suite, enabling them to manage IT environments based on intent rather than having to manually manage every process themselves.
At the core of those update is the Capacity Analytics Engine, which is being added to version 6.7 of vRealize Operations. Mahesh Kumar, senior director of product marketing for cloud management at VMware, said the capacity analytics engine makes use of Autoregressive Integrated Moving Average (ARIMA) techniques to forecast future system events based on historical data points that identify a long-term trend of sudden spikes in usage. Armed with that data, vRealize Operations can now provide cost analytics that include advice on ways to adjust application size and reclaim idle resources on either a public cloud or inside a local data center. IT organizations can also run “what if” scenarios, Kumar said.
On top of that capability, VMware is also now adding predictive analytics that enables IT organizations to define a set of intents such as workload balancing to reduce software license cost, prioritize one set of application to maintain a service level agreement (SLA) or make sure certain workloads only run on servers that have been shown to comply with a specific regulation or mandate. Tighter integration between vRealize Operations 6.7 and vRealize Automation 7.4 then can be continuously employed to automate DevOps processes in a closed-loop manner at higher levels of scale without having to add additional headcount, Kumar said.
In addition, the latest version of VRealize Automation includes more than 120 free, curated blueprints and OVF (Open Virtualization Format) files out of the box. VMware has teamed with Bitnami to add 20 new blueprints and 100-plus new OVFs of applications and databases including GitLab, Hadoop, Jenkins and MongoDB to speed application development and deployment. VMware also updated the Custom Form Designer and added multitenancy capabilities.
Finally, VMware vRealize Operations is sporting a new user interface and integrates with VMware Wavefront monitoring software.
Longer term, Kumar said vRealize Operations will be where VMware applies machine learning algorithms to enable artificial intelligence (AI) models to automate the management of VMware environments.
Collectively, Kumar said the VMware vRealize Suite moves IT organizations that have standardized on VMware in both their local data centers and in the public cloud much farther down the DevOps maturity curve. In effect, VMware vRealize Suite creates an “easy button” for managing complex VMware environments spanning multiple clouds, Kumar said.
VMware is also betting that AI models will make it a lot easier for organizations to make a cultural adjustment as the management of compute, storage and networking becomes more unified. It’s not clear how long it will take for those machine learning algorithms to manifest themselves in VMware management software. But it’s clear most IT organizations running VMware soon will need to rethink who is responsible for running what inside an IT operations team.
The DevOps community is rapidly building tools and processes to automate more of the DevOps workflow. One key is to put engineers on the front line who can find root causes of problems and automate more of the common tasks. Here are the two key areas to focus on when automating operations processes.
Automate Your Most Expensive Tasks First
The first step is to do some analysis on where the team spends most of their time. You will want to know if they’ve done a good job around the tickets and how many requests of what type have they gotten.
After analysis, one company found it had more than 500 tickets a day for user access requests from users who were locked out of their account. This is a common problem, especially in midsize to large companies that are doing user management or account management and have very manual processes for onboarding and offboarding.
There’s a ton of money that can be saved by automating these processes. And there’s a whole host of tools for everything from full orchestration systems to pipeline stripping to make it possible. But the first step is to analyze where time is being spent on the repetitive, easily duplicated tasks and automating those.
Automating Monitoring and Self-Healing
There are a lot of different tools out there, and the amount of data that’s coming together is enormous. With the emergence of big data and machine learning, we have the ability to put all the different points of monitoring into one system and bring all the data together to provide a single picture of how everything is performing.
Data sources can include anything from systems monitoring tools to application performance monitoring tools to social media feeds such as Twitter and Instagram. In addition, it can and should include data from your IT service management (ITSM) tools. Once you’ve got that you can correlate your performance data with your incident data to actually predict when a problem is more likely to occur so you can fix the root cause or the underlying issue before it impacts your users.
There’s some really neat stuff that can happen with automation, whether that’s just a simple script that can save you time and money or advanced machine learning to actually predict issues and address them in real time. Take the first step to make your organization more efficient and effective.
The successful application of machine learning for artificial intelligence (AI) requires several contexts to be addressed, primarily the technical, data and business considerations.
The Technical
From a technical standpoint, in building an AI-ready framework there are choices to be made such as which machine learning framework to select, the type and structure of learning function and the style of training algorithm to use. These decisions are normally based on the skill, experience and judgment of the computer scientist. A practitioner should be able to coordinate these aspects and focus on business outcomes without overly burdening the line of business with technical detail. The exception to this is when core design decisions in some way will impact business outcome.
The Data
Equally as critical to the efficacy of machine learning is the data available for use in development and training. A deployed algorithm is a function of the data inputs it received during training and testing; these inputs should be consistent and representative of the scenarios that will be subject to inference by the algorithm. It is difficult to accommodate so-called black swan events, as typically these require a level of adaptiveness beyond present machine learning technology.
However, we can still hope to achieve an effective and enriched algorithm that is appropriately responsive to real-world events by collecting and building data sets and training libraries that are truly representative of the environment in which the algorithm will operate. Ideally, machine learning data should include as many data outliers as may occur naturally within the real world—we should not be improving or self-selecting data, though granted most applications will use some form of normalization.
Speaking to data scientists and machine learning specialists, it is clear as individuals they have the technical skills and access to the core technology they need to do their jobs. However, more often than not it is access to and availability of real-world, suitably random data that is one of the biggest and most time-consuming challenges. Addressing this requires good engagement with the line of business, as access to sensitive data requires negotiation. But as matter of course, any business that utilizes data analysts to help inform critical business decisions and wants to benefit from machine learning should take the necessary steps to ensure it offers accessible and open data services to their machine learning teams. Having an open, accessible and agile data architecture is an essential prerequisite of being able to deliver an AI/machine learning-ready infrastructure.
The Business
On the business side, rather than address the how and what questions of technology and data, we also need to ask why. Certainly, as with any IT project, business does not want to engage machine learning or AI for its own sake. It is important to understand the business goals and seek new technology that will align with those objectives. This is where the application of DevOps techniques can greatly improve business outcome.
By aligning software development with software operations, more emphasis can be placed on what is important to the business while also helping to support the effectiveness and usability of new machine learning functions. In their deployment and use, these new applications should also exhibit many of efficiency features of modern IT technology. Thus, we expect new deployments to be able to make effective and efficient use of container technology. And where a particular function is infrequently required, how much more efficient is it to use a serverless function, perhaps with the data resident on private storage, and the function being called on demand in the cloud.
This type of capability is now readily available to machine learning thanks to the advent of virtualized GPUs provided by a choice of cloud services. This ability to dynamically and adaptively use infrastructure aligns well with the capabilities DevOps practitioners, who for a long now time have been used to managing the deployment of continuously developed releases of code. This agility, driven by the need for adaptability and innovation is very much what is required for successful machine learning and AI. But underpinning all of this is the need to maintain the conversation with the business around the open, yet appropriate accessibility to data.
In all the hype over AI, let’s concentrate on the real value it can provide, especially for busy IT Ops teams.
2018 is already shaping up to be the year when AI and machine learning (ML) go mainstream—at least in terms of public awareness, if not yet of actual usable products. As I write this, the annual CES event—the show that is, Prince-style, “formerly known as “The International Consumer Electronics Show—is in full swing. The usual suspects are out in force, with Amazon’s Alexa getting embedded in more and more products and Google also showcasing its own consumer AI capabilities.
Of course, much of what comes out of CES is frivolous, but there are all sorts of other applications of AI and ML. The problem is that whenever there is this much hype, disappointment and disillusionment are not far behind. Gartner’s famous Hype Cycle model can help here.
By Jeremykemp at Wikipedia, CC BY-SA 3.0
The conventional concern is that overly optimistic projects initiated at the Peak of Inflated Expectations risk failing, leading to excessive caution and fewer projects starting during the Trough of Disillusionment, before a more realistic understanding sets in and the technology achieves true mainstream adoption. Due to the nature of AI and ML, though, there is a second failure mode that is far more insidious.
AI and ML generally operates as a black box. Results are provided, very quickly and without the need to develop deterministic rules, but it is generally not possible to determine why a specific result was generated or troubleshoot failures in detail. As these techniques are used in the real world, potential problems that might not have been obvious in the lab begin to appear. If your smart fridge misreads the expiry date on that chicken (and you don’t think to check yourself), you might be in for an uncomfortable few hours. If your smart thermostat decides for some reason that your house should be as hot as physically possible, you’re still on the hook for the heating bill.
Those are the “nice” failure modes. If a facial recognition algorithm was trained on a data set that didn’t have the right racial balance, you might wind up with some of your employees not being able to get in the door, or some of your customers not being able to unlock your device, or whatever.
Don’t Throw The AI Baby Out With The Hype Bathwater
So should we just forget about this AI nonsense? Far from it. In fact, I am personally convinced that AI is very much a key part of our future. It just needs to be used in the right way.
I mentioned training data above, and that is the key. AI systems need a lot of good data, and they need to be trained to make good assessments. Out there in the real world, both sides of that problem are actually very hard. Gathering enough data, making sure that the data gathered are actually representative of the problem space, dealing with privacy issues around personally-identifiable information, and formatting the data correctly for ingestion by your machine learning system are already not trivial issues. However, even assuming that you have successfully dealt with that hurdle, the initial results will not be very good or reliable. Intelligent assessment by humans who understand the domain that you are training for is key if your deep learning model is not to go off the rails.
Come Up to the IT Ops Lab
Seen in these terms, IT Ops is an ideal domain to which to apply these techniques. First of all, IT infrastructure already generates huge amounts of event data – Big Data, if you insist – and all of those events are already, by definition, formatted to be easily machine-readable. Sure, there are many different formats out there, from SNMP, via various proprietary product-specific formats, to modern REST APIs, but it turns out that computers are pretty good at translating information from one representation to another.
Once the event data are in the AI system, it can apply whatever algorithms to try to make sense of them: which events are even significant, which ones are related to each other, what other relationships might exist, and so on. When it comes to evaluating the results, again, IT Ops already has people, tools and processes working on precisely that question: is this a real issue, and if so, what to do about it — or if not, why not. This evaluation can be fed back to the learning system to improve its results for next time, iterating over time to provide ever more valuable and reliable results.
This learning can even be applied to the evaluation process itself, watching what users find most interesting and valuable, and extrapolating from that to present the most significant data first, or looking for patterns in human interaction to assemble the most effective teams to work on particular types of incident.
This is what AIOps is all about. Don’t assume that the sort of consumer-grade AI that you see at CES is all that is going on in this field. AI has huge potential to help streamline enterprise IT Ops, by presenting human specialists with actionable events, helping them collaborate more effectively, and learning and improving over time.
Once the CES hype has died down, take a good look at AIOps and start to think how it can fit into your plans for 2018.
This is the second installment of a two-part Q&A series on artificial intelligence (AI) with AJ Abdallat. It covers AI’s potential significance to computing; how AI affects jobs; the employee specializations needed to fuel AI; big data; and cultural challenges in the enterprise. Be sure to check out Part 1 as well.
AJ Abdallat is a serial entrepreneur who, from 1998 to 2011, worked with NASA’s Jet Propulsion Labs (JPL) and Caltech on a series of tech incubation startups. The goal was to commercialize NASA technologies, many of which were artificial intelligence-related.
In 2014, Abdallat co-founded and became CEO of Beyond Limits, a startup that secured licenses to NASA AI technologies for use in commercial applications. Beyond Limits currently holds exclusive licenses to 42 blocks of sophisticated intellectual property developed through NASA R&D investment, a $150 million AI technology head start under the aegis of Caltech’s Office of Technology Transfer and Corporate Partnerships program. Abdallat’s company is a leader in cognitive technology, which goes beyond conventional AI, binding deep learning and machine learning tools with symbolic AIs to emulate human intuition.
We interviewed Abdallat while researching the story, “AI Delivering on the Business Analytics Promise.” With AI poised to potentially revolutionize business, Abdallat’s career, which blends AI science and commercialization of the technology, makes his insights fascinating.
devopsy.kinsta.cloud: How does the advent of AI as used by enterprises compare to earlier significant computing milestones like networking, the internet, or cloud computing?
Abdallat: Part of me wants to quash as much artificial intelligence speculation as possible. The hyperbole. But frankly, I think AI is potentially more transformative than the other things you mentioned, which are just infrastructure. AI is starting to help us pose the questions that we ultimately want to answer. What’s unique about AI is that it’s an introspective exercise. Every new success for AI is giving us some insight about how we work.
AJ Abdallat is co-founder and CEO, Beyond Limits, an artificial intelligence firm. He’s an engineer and entrepreneur.
As we integrate AI, the trend may not be that transformative initially. Improvements will likely be focused on doing it faster, bigger with lower downtime and more efficiency. But what comes next is thinking about all the humans involved in that process, and what they touch and how they work. I think once AI starts to integrate with that activity, and act at that level within the enterprise, this will truly change how we think about what a business is meant to do. I’m a little biased in this, but you might go so far as to say we are entering the era of the cognitive corporation, where man and machine at all levels of the enterprise will operate in a synergistic fashion to solve problems, each learning from the other.
If you think about most business processes and most problems that humans find interesting to solve, they are heterogeneous. They move through a pipeline of different specialists that look at separate pieces or provide specific oversight to balance specialized objectives. If I can couple my engineering and sales activities at the enterprise level, and use AI to mediate the trade-offs of both of those objectives, and I have these people processes synchronized with machine processes, it gives me the ability to learn how these two potentially juxtaposed business ends are coupled, where they’re divergent, and what kind of switches I can flick or knobs I can turn to make the net positive of these two forces all the greater.
devopsy.kinsta.cloud: Is artificial intelligence intended to replace humans or help them do their jobs?
Abdallat: In some places, as with any kind of automation, you will see some jobs go away. But the labor that was there before finds a new role working with AI, and I think those new roles, especially at the level of knowledge work, will be very interesting.
As good as artificial intelligence systems are, they’re not going to set our objectives. They can move toward them, and think outside the box, which is one of things that we work hard to do at Beyond Limits. But being able to deal with the nuances of problems is still something that humans excel at. So, finding the things that are worth teaching to AI, to learn and to figure out, that’ll still always be a job in the hands of humans. I imagine that’s also when it gets kind of cyclical. I’m going to learn something, the machine’s going to discover a new pattern, we’ll optimize an approach that I didn’t think could be optimized to such a degree. That’s going to give me new questions, which makes me give it new training data, and the process continues like that.
devopsy.kinsta.cloud: What types of software engineers are now or soon will be in demand to support AI, machine learning and this new industry?
Abdallat: One of the things we’re starting to see is a change in job definition. You probably hear about data scientists and machine learning scientists. What we’re going to see more of is taking that science and flipping it with engineer. There’s a ton of stuff in the laboratory that works and is great. And scientists can write papers and come up with nice coefficients for polynomials that solve problems. But at the end of the day, there’s an infrastructure that makes all this work. So, we will probably begin to see new job titles emerge, like data engineer. A data engineer is maybe the bridge between the work that the data scientist does and the infrastructure in which his data lives. He or she is much more of a software engineer but has some knowledge of the data science workflow and algorithms germane to data and AI work.
Data engineers are becoming more important, and we’re finding they fit somewhere between IT and the business. I think that’s important. It’s the same with machine learning people. For example, the actuary of the future will simply be a data scientist who works in insurance. Systems analysts will gradually become systems data scientists. Of course, to make all this stuff work securely, we’ll probably see a ballooning of the specialized security people to support the work that goes into the systems engineering. Unfortunately, the world is more adversarial than we would like.
devopsy.kinsta.cloud: Does AI present a cultural challenge to enterprises?
Abdallat: Earlier big data solutions and the fact that AI looks promising for big data have paved the way. AI asks for similar machineries, although it can be used in smaller footprints when available. Culturally, some IT organizations are more progressive than others. Some like to play with new toys and would be stoked to get this down, assuming they have the budget, and AI may be enough to motivate them to allocate budget.
In other cases, there may be entrenched opposition to change. But it’s such an exciting field that whenever you start talking about artificial intelligence, you’re going to get buy-in. Some healthy skepticism is a good thing too. You can try to channel it into constructive ways of alleviating the concerns that people bring up, and assess whether they’re just nay-saying the next new thing or they’re aware of a valid institutional or physical barrier to success.
There’s a revolution underway in data analytics that could create a competitive advantage for companies that jump on it. C-suites, take note: AI-powered business analytics is finally delivering future-facing data points that CEOs need to guide their businesses to the sweet spot.
The artificial intelligence (AI) technologies machine learning and deep learning are being used today to power through mountains of big data, much of which is previously inaccessible unstructured data such as text, video, customer reviews and customer feedback. Data scientists analyze the trends the machine learning uncovers, resulting in human-vetted, AI-derived, predictive analytics that provide strong decision support.
Companies using AI-enabled business analytics are seeing positive results, yet most believe we’re just at the tip of the iceberg in utilizing AI’s capabilities in decision-making. Indeed, almost 70 percent of the CIOs who responded to a recent survey said they believe decisions made by machines will soon be more accurate than those made by humans.
“Machine learning and deep learning have fundamentally changed enterprise data analytics,” said Tom Wilde, CEO of machine learning solution provider indico. He pointed out that a large majority of data is unstructured; in fact, according to some measures, as much as 80 percent of enterprise data today is unstructured. “In many cases, the unstructured data is where the real signal can be found,” and, as such, AI identification and categorization techniques can be used to develop analytics that could be used to improve a company’s product, predict future sales, identify cross-selling opportunities and reach new markets.
“Machine learning supercharges business analytics and data science,” said Scott Zoldi, chief analytics officer at FICO. Machine learning algorithms explore relationships and data that even the best data domain expert may not think to explore, but, properly used, “Machine learning augments data scientists to make their discoveries seem superhuman. Data scientists that use machine learning will ultimately drive differentiated business value for their organizations.” In other words, AI is like using newly invented night-vision glasses for deep dark data—it should give a company a tactical advantage.
What C-Suites Should Know about AI and Analytics
AI and business analytics may seem like a panacea for what ails a business. But it’s not a plug-and-play technology—AI requires constant care and feeding to churn out the trends behind the data. Here are three things the C-suite must understand before adopting AI as a business driver:
AI in its current form is unlikely to replace a human being. It has transformative power, yes, “but AI is going to further a lot of progress that’s already in place, not necessarily upend everything,” said AJ Abdallat, CEO of AI firm Beyond Limits, an AI company. AI will be used to create great things in years to come, but it’s not a magic bullet, it’s not infallible and it requires humans in great measure today. People should not be concerned about their jobs, nor should there be any expectation that AI might save money on personnel. These cultural concerns, along with a not uncommon resistance to change, are one of the barriers to entry.
If your company punted on digital transformation, you’ll probably need an outside vendor to help you foray modestly into artificial intelligence. “The sooner you can modernize your digital infrastructure, the sooner you can bring AI in to address your [larger] business issues,” said Abdallat. In many ways, the ability to employ machine learning and AI may wind up being looked at as one of the main positive outcomes of digital transformation. Working in concert, these technologies are probably your best protection against sudden disruption by a digital-native startup with good ideas.
“Too many CEOs delegate AI to their innovation teams, who conduct small-scale experiments. It’s important for CEOs to bring their own AI vision to their organizations,” said Gurjeet Singh, executive chairman and co-founder of Ayasdi. CEOs need to apply AI to the most pressing problems facing their businesses. “The world is going to change far faster than they can grasp. They need to set course immediately.”
Adopt AI Soon
One thing that has become clear as more companies join the AI fray, either as purveyors or consumers of the technology: The time for digital infrastructure improvements and adoption of AI is now. That said, the scope of enterprises that have already adopted AI is comparatively narrow. Companies currently using artificial intelligence to explore big data include the obvious: Amazon, Apple, Citi, DoCoMo, Facebook, Google, HSBC, IBM, Lockheed, Microsoft, Mt. Sinai, Tesla and Twitter. Many other companies are experimenting with AI in one facet of their business.
How significant a trend will the use of AI by enterprises be? Is it at the scale of networking, the internet and cloud computing? “AI will have the same or greater impact on businesses for the next 20 years,” Wilde predicted.
“Part of me wants to quash as much of the AI speculation as possible. The hyperbole,” Abdallat added. “But frankly, I do think that AI is potentially more transformative [than the internet, cloud etc.] … I know I’m a little biased in this, but we might be entering the era of the cognitive corporation, when man and machine at all levels of the enterprise will [cooperate] to solve problems, each learning from the other.”
At its core, DevOps is meant to touch an entire organization; however it’s heavily weighted in the software development life cycle (SDLC). That’s why DevOps is going mainstream in 2018. Companies are making the business benefits of DevOps a top priority—it’s all about delivering value to the customer faster through maximum automation of the release cycle/processes. And that begins with organizations realizing that to mature their DevOps processes, they need continuous testing as part of their continuous integration practice.
Cross-platform Engagement and IoT Will Mean More Devices—and More Apps
With organizations shifting processes and innovation to a mix of digital technologies—a concept known as digital transformation—consumers are engaging with different apps across various platforms 24 hours a day, seven days a week. Add in the rise of internet of things (IoT) devices, which allow for seamless transitions across smartphones, TVs, tablets and other devices. Now is the time for optimum user experience (UX).
In 2018, industries such as financial services, health care, retail and automotive will fully embrace IoT. We may also see new types of applications replacing current ones, as consumers and developers embrace the constantly evolving innovative capabilities coming in 2018. However, until the C-suite implements the tools needed to keep up with the growing demand for cross-platform upkeep and testing, other industries will experience slow adoption.
A critical step in delivering flawless UX is testing, and lots of it. Never has there been more to test, measure and develop than in this digital revolution we are seeing today.
Frequent New Releases Will Require Even Faster Updates
As the number of platforms consumers engage with rapidly grows, so does the pressure for developers to ensure apps are running as they should at all times. Same goes for the pace of cloud adoption: Cloud test labs will need more sophisticated capabilities, meaning more platforms and blind spots which require continuous optimization to ensure market readiness. This need for faster turnarounds on update releases and bug fixes will multiply in 2018.
To maintain and grow customers and profitability, CIOs must recognize the importance of empowering developers with the tools and time to implement continuous testing across the SDLC. Tools, such as those powered by automation and the cloud, increase efficiencies and free up time developers previously spent on manual quality-checking, allowing them to ensure the apps they are producing are keeping up with consumer expectations. If users feel their app is not delivering on the experience they want, they will delete it and move on to the next competitor, which has a direct negative impact on the company’s bottom line.
Outdated Technologies Will Make for Even Slower Updates
There is a suite of new tools being introduced to the market for all stages of the SDLC in the hopes of keeping up with the heightened speed of testing. As many DevOps environments struggle to adapt their pipelines to these new tools, many are powered by emerging technologies including machine learning and automation. In 2018, developers should ditch outdated infrastructure and processes in favor of more modern architectures, including implementing microservices.
Modernizing pipelines will allow developers to design in next-gen engagement methods such as artificial intelligence (AI), augmented reality (AR) and virtual reality (VR) without compromising velocity.
AI Will Experience Its Shining Moment (for Some)
In the coming year, AI will start to move from a theoretical concept to a commonly applied technology, especially within the DevOps space. However, developers first must understand what they want AI to help them accomplish—and how—within the SDLC and across the DevOps pipeline. One logical place to start is figuring out how they can best leverage it to analyze test automation strategies.
The value of a well-oiled, or not-so-well-oiled, DevOps process can make or break an organization. As the new year begins, be sure to include the IT leadership in these conversations to ensure developers are getting what they need, when they need it, to keep business moving as usual. IoT, AI and the cloud are just a few things to keep in mind as the industry moves to implement the latest tools in 2018.
About the Author / Eran Kinsbruner
Eran Kinsbruner is the lead technical evangelist at Perfecto and the author of the “Digital Quality Handbook.” He is a software engineering professional with nearly 20 years of experience at companies such as Matrix, Qulicke & Soffa, Sun Microsystems, General Electric, Texas Instruments and NeuStar. He holds various industry certifications such as ISTQB, CMMI, and others. Eran is a recognized mobile testing influencer and thought leader. He is also a patent-holding inventor (test exclusion automated mechanism for mobile J2ME testing), public speaker, researcher and blogger. Follow him on Twitter, connect with him on LinkedIn and check out his professional mobile testing blog.
Thanks to the rise of multi-cloud computing, the challenges associated with managing DevOps are becoming exponentially greater. Most IT organizations are not likely to be 100 percent sure at any given time which application workload is running where.
To address that issue, Morpheus Data has added a Cross-Platform Discovery module to its Unified Ops Orchestration platform that employs machine learning algorithms to determine not only what applications, virtual machines and containers have been deployed on-premises or in a public cloud, but also how much capacity, memory usage and power consumption each is using and their and overall performance.
At the same time, Morpheus Data is adding integration with Github and Jenkins continuous integration/continuous delivery (CI/CD) environments as well as support for both Docker Swarm and Kubernetes clusters.
Finally, Morpheus Data has added connectors to IBM Cloud, Upcloud and HPE OneView services and been certified as a ServiceNow partner. The Unified Ops Orchestration platform already supports Amazon Web Services (AWS), Microsoft Azure and Google Cloud Platform.
Brad Parks, vice president of marketing and business development for Morpheus Data, notes that as IT environments become more distributed in the age of the cloud, new approaches to change management will be required. The Unified Ops Platform provides an opportunity to consolidate change management processes within the context of a single control plane.
While more IT organizations are making use of multiple cloud services, most are managing each IT environment in isolation. Over time, the lack of unification of the control planes for each of those environments conspires to increase the total cost of ownership for IT. Morpheus Data is making a case of for unifying the management of multiple platforms running on-premises or in public cloud instead of requiring IT staff to master multiple management tools.
That single management plane, Parks says, also makes it much easier for IT organizations to take dynamically employ multiple cloud environments. Price points for cloud services are liable to change at any moment, but it doesn’t necessarily follow that the same type of workload will continue to be deployed on the same cloud. At the same time, the characteristics of a workload may have evolved to the point where it doesn’t make financial sense to deploy it on a public cloud. A common management platform makes it possible to move that workload with the least amount of disruption possible.
Most IT organizations today are concerned about getting locked into a cloud service provider. Being able to demonstrate an ability to switch between cloud service providers gives IT organizations an opportunity to negotiate cloud service contracts from a position of strength. Most cloud service providers are counting on the fact that the cost of switching from one provider to another is simply too high. Integrated DevOps processes wrapped around a common management plane, however, will make it clear to all concerned that the platform provider serves at the pleasure of the internal IT department, not the other way around.
Log analytics is all the rage these days among IT and security professionals trying to discover the root cause of any performance issue or security breach. But there’s little consensus about how best to go about analyzing logs. Many IT organizations have made major investments in commercial platforms such as Splunk, while others prefer to rely on the less expensive, open-source tools Elasticsearch, Logstash and Kibana, collectively known as the ELK stack.
As a provider of a cloud service based on the ELK stack, Logz.io has been providing organizations that prefer to rely on open-source software an alternative to deploying and maintaining their own ELK stack. Now Logz.io is extending that service to include Logz.io Application Insights, which employs machine learning algorithms to create a model of normal IT operations that can be used to more easily identify anomalies, and Logz.io Data Optimizer, which helps identify which logs are worth keeping long-term.
Fresh off raising an additional $23 million in capital, Logz.io CEO Tomer Levy says his company is building up a pool of machine data to drive advanced analytics fueled by machine learning algorithms. Those insights then can be shared proactively with other Logz.io customers to head off production issues before they even occur. At present, Logz.io has more than 400 customers.
Levy says the Logz.io Data Optimizer should enable IT organizations to substantially reduce their costs. Most providers of log management tools charge based on the amount of data collected. By helping organizations identify extraneous log data, the cost of implementing analytics of machine data drops considerably, Levy says.
Logz.io Application Insights, meanwhile, provides a simple way to identify the last changes made to the environment that might have created a performance issue or introduced a security vulnerability.
Levy says Logz.io, via application programming interfaces (APIs), can be integrated with multiple continuous integration/continuous development (CI/CD) platforms, including Jenkins. In fact, log analytics is now a fundamental element of any mature set of DevOps processes, he notes.
Platforms for analyzing machine data is one of the fastest growing sectors of enterprise IT. New research from Zion Market Research forecasts that the IT operations analytics market is expected to grow on a 23.7 percent compound annual basis to reach $9.2 billion by 2020.
Most DevOps issues only take a few minutes to resolve. But identifying the root cause of an issue can take weeks, sometimes even months, to identify. In more advanced IT organizations, centers of excellence around log analytics are now being created to identify not only potential IT and security problems, but also issues impacting, for example, the amount of revenue being generated via a specific application.
It still may be a while before log analytics become pervasive across the enterprise. But every event in the IT environment generates a log, so the real challenge and opportunity is correlating all those log events into something that manifests itself as a truly actionable intelligence.
IPsoft wants to put a digital face on the future of IT operations that will be infused by artificial intelligence (AI). Its recently launched 1Desk service desk combines the Amelia digital assistant with IPsoft’s Apollo virtual engineer technology to automate multiple backend processes.
Allan Anderson, director of enterprise solutions for IPsoft, says 1Desk effectively combines two longstanding areas of IPsoft research to provide IT organization with an ability to manage IT at scale using digital personal assistants and virtual engineers.
IPsoft has been providing managed services based on its Apollo AI framework for several years. The launch of 1Desk creates an opportunity for IT organizations to not only employ that AI framework on their own, but also combine it with a digital assistant technology christened Amelia that IPsoft has trained to handle a variety of technology and business process requests.
Anderson says the virtual engineer AI technology developed by IPsoft make use of machine learning algorithms to continually monitor how humans complete a task. Armed with that data, those virtual engineers can then incorporate that information into their knowledge base to learn a process or fine-tune an existing one.
A continuing shortage of IT skills in key areas is significantly reducing resistance to AI technologies being applied inside IT organizations—AI technologies may enable organizations of almost any size to operate IT environments at unprecedented levels of scale while minimizing human errors. IPsoft, says Anderson, expects virtual engineers to be employed as a digital labor force capable of performing tasks such as identifying and killing a long-running query to configuring and executing runbooks across the most complicated data center environments. To facilitate the automation of the those processes, IPsoft has developed a Digital Labor Studio tool that organizations can use to create and train a virtual agent.
There’s no doubt that many of the tasks associated with managing an IT environment are repetitive. In fact, any repetitive IT task is arguably a ripe candidate for automation. How those advances will impact DevOps processes remains to be seen. It’s not difficult to imagine that each development team will be assigned its own digital assistant. The role of the IT operations staff then would evolve into one where a smaller group of people train and manage a pool of digital labor.
IT professionals are generally used to the idea that IT will be used to automate a task that used to be part of someone else’s job. But now IT is being used to automate many of the tasks performed by IT professionals. It may take some time for them to first accept that idea and then evolve their skills accordingly. There will also be a need for some level of human involvement when it comes to IT. But instead of thinking in terms of managers and administrators, perhaps the brave new world will be one where everyone in IT is now the manager of their own digital force of labor.
Eran Kinsbruner is the lead technical evangelist at 
