devopsy.kinsta.cloud

Category: AI

Top 10 Emerging Technologies Enterprises Should Know
The momentum behind the adoption of new technologies underway is no less than astounding.

Those enterprises that not only adopt, but learn to master, emerging technologies will be the clear winners in the years ahead as they learn their customers’ needs better, rapidly develop new and successful products and services, optimize their use of business technology, improve supply chains, increase margins and stay a few steps ahead of the competition.

Recently the World Economic Forum released its annual list of technologies it thinks are ready to “break through” this year, which generally means start witnessing wider adoption. Some of the breakthrough technologies this year include very powerful batteries, what is deemed “socially aware” artificial intelligence (AI) and improved solar panel technology.

According to a statement released by the World Economic Forum, the Top 10 2016 Emerging Technologies list, compiled by the Forum’s Meta-Council on Emerging Technologies and published in collaboration with Scientific American, reveals the technological advances the group believes could improve lives, transform industries and safeguard the planet. The list also serves as an opportunity to discuss the any human, societal, economic and environmental risks the technologies may pose prior to widespread adoption.

“Horizon scanning for emerging technologies is crucial to staying abreast of developments that can radically transform our world, enabling timely expert analysis in preparation for these disruptors. The global community needs to come together and agree on common principles if our society is to reap the benefits and hedge the risks of these technologies,” said Dr. Bernard Meyerson, chief innovation officer at IBM and chair of the Meta-Council on Emerging Technologies.

Some important criteria in technology selection was the council members’ belief that 2016 would prove to be a deployment tipping point of each technology. That should be of interest to CIOs and technology leaders, too. And it means it’s time for tech leaders to start figuring whether and how implementation may or may not make sense.

The top 10 technologies to make this year’s list are:
1. Nanosensors and the Internet of Nanothings – With the Internet of Things expected to comprise 30 billion connected devices by 2020, one of the most exciting areas of focus today is now on nanosensors capable of circulating in the human body or being embedded in construction materials. Once connected, this Internet of Nanothings could have a huge impact on the future of medicine, architecture, agriculture and drug manufacture.
1. Next Generation Batteries – One of the greatest obstacles holding renewable energy back is matching supply with demand, but recent advances in energy storage using sodium, aluminum and zinc-based batteries make mini-grids feasible that can provide clean, reliable, around-the-clock energy sources to entire villages.
1. The Blockchain – Much already has been made of the distributed electronic ledger behind the online currency Bitcoin. With related venture investment exceeding $1 billion in 2015 alone, the economic and social impact of blockchain’s potential to fundamentally change the way markets and governments work is only now emerging.
1. 2D Materials – Graphene may be the best-known, single-atom layer material, but it is by no means the only one. Plummeting production costs mean that such 2D materials are emerging in a wide range of applications, from air and water filters to new generations of wearables and batteries.
1. Autonomous Vehicles – Self-driving cars may not yet be fully legal in most geographies, but their potential for saving lives, cutting pollution, boosting economies and improving quality of life for the elderly and other segments of society has led to rapid deployment of key technology forerunners along the way to full autonomy.
1. Organs on chips – Miniature models of human organs—the size of a memory stick—could revolutionize medical research and drug discovery by allowing researchers to see biological mechanism behaviors in ways never before possible.
1. Perovskite Solar Cells – This new photovoltaic material offers three improvements over the classic silicon solar cell: it is easier to make, can be used virtually anywhere and, to date, keeps on generating power more efficiently.
1. Open AI Ecosystem – Shared advances in natural language processing and social awareness algorithms, coupled with an unprecedented availability of data, will soon allow smart digital assistants help with a vast range of tasks, from keeping track of one’s finances and health to advising on wardrobe choice.
1. Optogenetics – The use of light and color to record the activity of neurons in the brain has been around for some time, but recent developments mean light can now be delivered deeper into brain tissue, something that could lead to better treatment for people with brain disorders.
1. Systems Metabolic Engineering – Advances in synthetic biology, systems biology and evolutionary engineering mean that the list of building block chemicals that can be manufactured better and more cheaply by using plants rather than fossil fuels is growing every year.
July 4, 2016
Machine Learning, AI Driving DevOps Evolution

Automation is the fuel that drives DevOps. Automating routine, repeatable tasks is one of the defining characteristics of DevOps culture. As artificial intelligence (AI) and machine learning improve, the scope and complexity of the tasks that can be automated increases, which raises the bar for all of DevOps. The humans behind the DevOps will be freed from even more mundane tasks and able to focus on more innovative and creative endeavors.

It makes sense, if you think about it. The trend from physical services in an on-premise data center, to virtualization, to cloud services, to DevOps has been a steady march of raising the bar for automation. The more applications and entire infrastructures can be developed, deployed, monitored and managed programmatically and through automation, the more efficiently and effectively an organization can function.

As tasks are offloaded to automation, humans can address other challenges. That focus inevitably leads to identifying other new tasks that can be automated. AI and machine learning represent a quantum leap in this evolution, however—with AI and machine learning, the automation can begin to identify and improve itself. When the automation can identify bottlenecks and inefficiencies and automatically adapt to overcome those obstacles, human input and interaction could become almost obsolete.

In a recent interview, Christian Beedgen, CTO of Sumo Logic, shared some thoughts about the impact of machine learning on DevOps. “Machines will enable greater productivity among teams and businesses through the advances that machine learning and automation can offer,” he said. “A necessary component to augment human contributions, machine learning will finally give humans the power to review millions of bits of data.”

Automating routine tasks is crucial, but there’s another factor that plays into the role that AI and machine learning have with the future of DevOps. The reality is that there are some things humans simply can’t do as well or as fast as machines—especially at scale.

Steve Burton stressed in a blog post that humans are capable of observing, digesting and interpreting only so much information at one time. Some feel like the Holy Grail solution is some sort of application performance management (APM) dashboard that can aggregate and correlate real-time data so human IT personnel can monitor metrics and identify issues manually. “This all sounds good in theory, except it’s completely unmanageable in reality when your applications have thousands of components, billions of metrics and hundreds of changes every day,” he wrote. “Humans can no longer cope with this complexity, they now need machines to do the leg work, process this Big Data and provide operational insight into what the f**k is going on.”

Beedgen agrees. He suggested, “Increasingly, businesses will adopt machine learning technology for its usefulness in augmenting human understanding of complex interaction and large data sets by uncovering the unknown unknowns.”

On some level this all sounds a bit scary. It seems as though the more we succeed at DevOps and the more DevOps evolves, the less humans are even a necessary part of the equation. Conceivably you could be so good at DevOps that you DevOps your way right out of a job or inadvertently launch SkyNet—or both.

That scenario is simultaneously feasible and far-fetched. I suggest that you don’t fear that future or look at it as machines stealing your job or taking over humanity. Embrace the future of DevOps driven by AI and machine learning and find the silver lining. Seek out the new frontiers and horizons, because there always will be tasks that the machines simply can’t do without us—and that is where innovation happens.

January 6, 2016
The Big Question

TEST

Here are some other ROELBOB’s you might like

A Unit Testing Dillema

The Pink Slip

Another Pink Slip

Hot off the Press – Other great devopsy.kinsta.cloud content

DevOpsSec: 1 in 16 Chances

CIOs Must Fight for Relevance in Face of Tech Storms

— ROELBOB

December 16, 2015
Avik Partner’s Grok APM Uses Machine Learning to Find Anomalies Early On

Machine Intelligence-Driven Anomaly Detection

Machine learning—the science of getting computers to take thoughtful action without explicit programming—is the basis of Machine Intelligence-Driven Anomaly Detection, which is the core capability in Avik Partner’s Grok APM product, according to Casey Kindiger, veteran enterprise software solutions provider, IT automation expert, and founder of Avik Partners.

“Thoughtful action means using the type of faculties that a human being would use to make a decision and to take an action,” says Kindiger, who has more than a decade of experience in designing and developing process automation solutions for companies such as JP Morgan Chase, T-Mobile, and Allstate Insurance. Grok learns from your dev and ops environments and takes thoughtful action to detect anomalies, all in the service of Application Performance Management.

Grok works by analyzing real-time performance data from apps and infrastructure, looking for behavioral bumps in an otherwise smoothly-paved DevOps highway of deployed services and underlying infrastructure.

Grok’s Pre-Event Inferences

“Grok uses common statistical techniques to learn a pattern over a selected period of time in order to build a dynamic threshold of the continuously changing underlying patterns,” says Kindiger. In this way, Grok can establish a sort of dynamic baseline from very noisy patterns, which is something that would typically take a number of person hours of involved statistical and predictive analytics to accomplish.

Grok’s machine intelligence leverages the ability to make inferences based on this dynamic baseline information like a human being would in order to reason out a logical conclusion. Grok does this to detect the slightest of anomalies based on the difference between what reasonably should have happened and what actually did happen. Using this approach, Grok detects subtle patterns of significance in streaming data despite the fact that the data has patterns that are continuously in flux.

But rather than detect when something has gone wrong at a point in time, which is not scalable or sustainable, Grok looks for minute patterns that appear well in advance of any issue that you would normally consider looking at with traditional APM mechanisms, explains Kindiger. Rather than send alerts based on threshold-based exceptions, Grok makes a prediction on what the next data point or the next series of data points will be and then looks at deviations from those predictions, considering whether to flag those as anomalies, according to Kindiger.

Currently, Grok is particularly tuned to analyze streaming data and is, according to Kindiger, data-point agnostic. “We don’t need to teach the system what the context of a particular data point is. The system learns the context of the data over time and the subtle nuances of the pattern that produces over time, and creates the model based on that,” says Kindiger.

Watch Out for That Initial Two Week Delay

If Grok does everything that Avik Partners’ Kindiger hypes that it will, DevOps teams will soon be able to catch and act on infrastructure issues before they grow big enough for end-users to notice.

But Grok does have one wrinkle in its APM armor: it takes time for it to really learn a new customer system and get good at detection. Let’s assume that Grok’s machine learning causes it to get better at detecting anomalies in a given set of infrastructure over time. That means that there’s a point, at the beginning, given a brand new customer when Grok only starts to learn. At that point, Grok isn’t very good at knowing what’s anomalous compared to what’s normal for that infrastructure because Grok has no foundation for that particular system.

“Grok will take some time to learn the nuances of when to escalate an issue. We’ve learned from experimentation that it gets pretty good after ingesting two weeks’ worth of data,” says Kindiger. Because Avik Partners initially targeted AWS, it has taken two weeks of CloudWatch data and fed that to Grok to provide it with some backdrop so that when Grok goes live, it won’t be learning this environment entirely anew.

November 3, 2015
How machine learning could head off security quality failure

With so many players in the mix from operations to developers, from PMs to technical leads, to Q&A, most of their activities are automated. Systems record all their email, Wiki, and chat conversations and other notations about these activities. This gives the devops community the opportunity to learn from that data in order to improve future projects.

“We have access to a massive amount of data that can in total define what the process is that’s going on and define it in-flight in real-time,” says Aaron Cois, Researcher, CERT Division, Software Engineering Institute, Carnegie Mellon University. Through exploratory research, Cois is learning that by using machine learning tools, enterprises can analyze security activities in the development process based on data from issue trackers and commit logs.

Valleys in Topic Density Point to Software Development That is Missing the Mark

Natural Language Processing, a subset of machine learning, uses systems to process text in unstructured data and to perform topic detection, determining whether or not a portion of the text belongs to a given topic category. “In our work, we’ve taken text from issue trackers and used machine learning to classify each issue as security-related or not security-related,” says Cois.

This has practical applications for detecting the course of a project and steering it in the right direction. If the PM expects the developers to raise the quality of security for an iteration of the software package, he can monitor the chatter among developers about the issues they are adding to the issue tracker or he can monitor the comments on the code.

“By using machine learning to classify issue tracking or source code commit messages as security-related, we can see a reasonable measure of the amount of work that’s being done to ensure the security of the software system,” says Cois.

If the PM can determine based on a shortage of security chatter that in actuality not much security work is going on, he can ask why, what is the challenge? “Is the security stuff more difficult? Did we do our due diligence to allow our developers to adequately address security?” asks Cois. Then the PM and the teams can work together to remove any roadblocks before any more scope creep or veering off course occurs.

Machine Learning for Physical Modeling and Trends Analysis

If the industry can build statistical models of what a healthy software project looks like as it progresses, then it should be able to determine whether a given project is on track and what risks it may face along the way.

Issue tracking data, commit logs, wikis, chat server logs, build/deployment logs—every type of data that can lead to a full picture of project activities—would form the basis of these models, according to Cois.

Through Natural Language Processing, machine learning can help determine from this data what the rates of creation and resolution of issues are and what the rates of commits and overall progress are for the software project.

Machine learning can help PMs look at how their project’s rates in these areas compare with rates of similar projects, which developed certain problems or turned out in certain ways. If there is a strong correlation between rates of projects that had known outcomes and rates in your project, then you might want to use that information to make corrections early on, in cases where the outcomes of the other projects were undesirable.

Over time, the models become stronger and their predictive value increases, leading to some strong correlations between current projects and the models collected. With the increasing density of the data available in models as they grow, even the weakest numbers in early models can become stronger, giving the model strong correlative value in areas where strong correlations did not appear at first.

“If I have collected both activity and outcome data on thousands of projects, and a large number of them are highly similar to the current project, I’ll have much more confidence in my predictions about the potential outcomes of the current project given the behaviors I am observing,” says Cois.

April 28, 2015
From Orwellian Nightmares To Unicorns: Why I’m All In On DevOps

Over the years I’ve helped literally thousands of customers gain efficiencies in their data centers. I’ve met and become friendly with a lot of IT pros but I’ve got to admit – I never envied them. Well, maybe I envied some of the views in NY and Jersey City. And some of the toys they got to play with over the years were amazing.

Nonetheless, I didn’t envy them because many of my friends in director and VP roles spent a huge percentage of their time fighting politics that resulted in Orwellian distortions of the truth. A couple of stories from the early 2000s might help illustrate my point:

1. Two guys pulling all-nighters delivering a $1.5B roll out of IP communications while 25+ project managers debate progress from 9-5

The headline pretty much captures it. This now too-big-to-fail bank, as well as their vendor, had brought in several project and program managers to ensure groups from facilities through supply chain through security and operations were “aligned.” I spent a couple of meetings getting to know all of these project managers before realizing that there were two guys working every night, all night, to manually, using fingers and spreadsheets, push and validate changes. Eventually those operators quit – and the project ground to a halt.

All the alignment, of course, was pretty much BS. At one point I mentioned in a meeting that our software would not run on the same server as the IP Communications System (Call Manager for those curious). I was corrected – “it isn’t a server.”

I responded with something like, “it is an HP DL something or other. Isn’t that a server?“

Answer: “No, if we call it a server then the server team will need to get involved and we don’t want the server team getting close to this project.”

You cannot make this stuff up. Servers = / servers. Got it?

2. Similar story, different bank – 1% of achievement of goal = “on plan” to outsourcer

The story starts when I was brought in by a huge vendor to a bank that is also too-big-to-fail. This bank also had a high profile project to get the bank over to IP communications. Keep in mind that this was after 9/11 and all the banks were looking up to the teams at Lehman that had been able to get Lehman back trading significantly faster in part because of their use of IP communications. So the project to get the bank onto a new approach to real time communications had CIO visibility.

One of the most surreal meetings of my career occurred when after having been vetted, tested, recommended, and designed into the solution I was granted an audience with the SVP handling the bank at EDS; I feel ok with naming EDS since they no longer exist and I’m sure this executive has retired. Like many folks at EDS he was former military and as an account executive he owned a north of $1B outsourcing contract.

I started off by saying that we can help you get back to schedule, as I know you have only done about 1% of the deployments originally planned. He looked me right in the eyes and said: “EDS is on plan, and we always will be.”

After some back and forth I got it – no matter what the facts on the ground were, he was never going to accept any culpability at all for the project delay. So missing the project time line by 99% was officially “on plan.” That’s Orwellian.

DevOps restores faith in humanity / IT

Compare any decent “DevOps” shop to the above stories and you can see why I’m so enthusiastic about doing my small part to help bring DevOps approaches to operations to a broader market.

The teamwork between not just Dev and Ops, but between everyone responsible for delivering capabilities to the organization, is so refreshing. Just last week, Google engineering blogged about using machine learning to improve operational efficiency of their data centers, with a focus on power consumption and so forth. The technology I get – we are headed down a similar path with deep learning and control theory. But what struck me more than the technology was the data they are using to draw inferences would often be siloed and tightly controlled by groups within IT at legacy companies precisely because it could be used to score points.

So DevOps entails more than infrastructure and code and collaboration. With DevOps, IT organizations move from fighting over terms like “server” and “on plan” that are often filled with well meaning and intensely frustrated teams towards an environment where cool stuff happens all the time. And IT done right – making sense of the world with software and hardware – is incredibly cool.

I’m betting on DevOps because I want the world to be filled with DevOps unicorns and because, like my friends that are operators within legacy environments, I cannot stomach more Orwellian obfuscation of the facts on the ground for political expediency. DevOps makes the world smarter and IT teams a heck of a lot happier. What’s not to like?

June 3, 2014