devopsy.kinsta.cloud

Tag: machine learning

  • Managing Complexity in Dev/Test Infrastructure Using Machine Learning

    Managing Complexity in Dev/Test Infrastructure Using Machine Learning

    Businesses are increasingly developing software solutions that can drive innovation and digital transformation. As the number of developers increases in the organization, the ratio of developers to IT also increases over time. The net result: more dynamic and complex dev/test infrastructure requirements and slower support from IT.

    Developers and testers often cannot get their hands on infrastructure environments in a timely manner to accelerate delivery of software applications to their customers. They have to deal with lack of timely support from IT and long waits for hardware resources and for provisioning and configuring of the environments.

    How does engineering IT manage this complexity and deliver more agility to their engineering stakeholders?

    Provide Self-Service Creation of Dev/Test Sandbox Clouds

    Engineering IT teams can increase the velocity of development and testing if they enable self-service ability for dev teams to quickly provision their own lab infrastructure and application stacks at a click of a button.

    Self-service, API-driven infrastructure is a fundamental requirement for enabling developers to write code (which can be done using their favorite programming language) and make RESTful API calls to the underlying programmable infrastructure. This allows them to dynamically and automatically manage initial deployments and configurations as well as manage ongoing, automated dynamic provisioning of infrastructure, autoscaling, monitoring and alerting.

    All this automation removes the confusion and error-prone manual steps for the entire application delivery process, including development, testing, staging and production deployments. This, in turn, accelerates software delivery and increases quality.

    Additionally, if self-service is available as a SaaS-based delivery model, it is easy to add more features and workflows very quickly without having IT doing a major upgrade.

    Breaking these silos improves collaboration between teams, accelerates velocity of software development, improves infrastructure utilization, increases overall operational efficiency and reduces costs.

    Eliminating Silos in the Dev/Test Infrastructure

    Silos are common in companies where various specialist teams (storage, networking, security) form fiefdoms around their respective functional areas. Silos impede velocity because they lead to complexity of operations, lack of consistency in the environment, and lack of automation.

    Automating across silos turns into to an exercise of custom scripts and lot of “glue and duct tape,” which makes maintenance and change management complex, slow and error-prone.

    Breaking these silos improves collaboration between teams, accelerates velocity of software development, improves infrastructure utilization, increases overall operational efficiency and reduces costs.

    A hyperconverged cloud design with a software-centric, scale-out architecture tightly integrates compute, storage, networking and virtualization resources and other technologies from scratch in a commodity hardware box supported by a single vendor. Also, companies can keep costs under control by leveraging scale-out cloud designs that make it easy to start small, grow based on demand, and stay close to the right size and customer demands.

    Additionally, development teams should have the ability to quickly deploy/clone/share complex multi-tiered application stacks between various teams and between development and testing, breaking the silos within the development organization.

    Automate Operations, Monitoring and Patching

    Engineering IT teams need to have complete visibility and control of their entire stack from the infrastructure up to applications. They need intelligent software to monitor the hardware and software stack, manage large-scale clusters and automatically handle routine but time-consuming and complex operations such as failure handling, patching, security updates and software upgrades.

    Running a service with traditional sysadmin teams who execute the above activities manually becomes expensive—especially if they operate in server, storage, networking and security silos—as the dev/test environments become more dynamic and the demand for more environments and projects grows.

    However, it is possible to use an intelligent private cloud platform that leverages hyperconverged scale-out designs, machine learning software and a SaaS-based operational console to reduce complexity and increase the agility of dev/test teams.

    Cloud-based monitoring and advanced analytics dramatically reduce the need for experts in different parts of the infrastructure, scale linearly as the size of the operation increases and cut operational complexity by 90 percent.

    Manage Resource Management Using Machine Learning

    Applying machine learning to infrastructure management means intelligent software could learn about operational patterns, anticipate capacity needs, raise alerts about security anomalies, self-monitor and self-heal your environment in the face of failures, intelligently apply security patches and automatically upgrade hardware and software systems without any downtime. The next generation of infrastructure management will be driven by advances in machine learning and artificial intelligence, where the infrastructure is able to basically “drive itself” with minimal user intervention.

    This will help engineering IT teams optimize resource usage and capacity based on current and future dev/test demand and also be able to better handle the availability and performance they can deliver to their engineering teams. A lot of efficient resource management comes down to capacity planning, utilization monitoring, right-sizing of workloads, demand forecasting and detecting zombie VMs and unused resources.

    Demand forecasting and capacity planning can be viewed as ensuring there is sufficient capacity and redundancy to serve projected future demand with the required availability. Capacity planning should consider organic growth, which stems from natural service adoption and usage by dev/test teams. Having intelligent predictive analytics and machine learning can help greatly with accurate forecasting, alerting and providing lead time for acquiring additional capacity.

    Better insights into how the infrastructure is performing can also help in fine-tuning performance of end user workloads. For example, an intelligent system that is monitoring a workload for storage performance might recommend using SSDs instead of spindles to increase the IOPS and improve workload responsiveness.

    An intelligent private cloud platform that uses hyperconverged scale-out designs, machine learning software, and a SaaS-based operational console can reduce complexity and increase the agility of dev/test teams.

    About the Author / Kamesh Pemmaraju

    Kamesh Pemmaraju leads product management, product marketing, and technology partnerships at ZeroStack. Kamesh is a seasoned product management/marketing leader with a strong engineering and consulting background. In his most recent product roles at Mirantis and Dell, he focused on delivering private and hybrid cloud solutions to enterprises. He holds a Master of Science in Computer Science from Indian Institute of Science. Connect with him on LinkedIn and Twitter.

  • Dev & Ops: These Are Reconcilable Differences

    Dev & Ops: These Are Reconcilable Differences

    The new world of IT requires a different approach from what worked in the past. As usual, the technology is the easy part, but without people and processes being aligned, it won’t deliver value. When evaluating how data science and machine learning can help manage highly dynamic IT environments, old reflexes may be actively harmful.

    By their nature and due to their experiences, Ops people tend to be a pretty (small-C) conservative bunch. Rule No. 1 of Ops is “If it ain’t broke, DON’T TOUCH IT!” and if we’re honest with ourselves, most of the rest of the rulebook is just variations and commentaries on that theme.

    One of the ways in which Ops people try to control their world is through automation. DevOps mostly looks to the moment of release and deployment, as that is where the handover to Ops typically occurs—and this is a big focus of the drive to automate. However, there is another part of Ops where automation is key, something that Dev is not always nearly as involved in; namely, day-to-day operations. Which alerts should even be sent, whom they should be routed to, what are the valid responses and so on—all these have been automated over the years, to a greater or lesser extent.

    Ops people have laboriously documented their architecture and spent a long time in meetings planning which information is relevant to share and who should respond to alarms. Much of this is implemented in various pieces of software, whether commercial, open-source, home-grown or that special grey area, where something that started out as a standard component has been so customized that it is effectively bespoke. Rules, filters and thresholds determine what action to take in which case.

    These approaches have been in place for years or even decades, but now they are starting to fail. Simply put, IT’s increased complexity and the ever-accelerating rate of change are outpacing administrators’ ability to keep up and reconfigure their management systems.

    The Coming ‘New Normal’

    New approaches are emerging to deal with the new normal, but as ever, the technology is the easy part. For new technology to be a success, people and processes need to be in sync.

    Right now, the emerging approach is to use data science and machine learning to process events, instead of the old deterministic rulesets and databases. The results speak for themselves, especially at scale and in highly dynamic environments. People building container-based or SDN architectures no longer assume that there will be a database of configurations that will always be up to date; quite the opposite.

    The new way is to throw out all the filters and rules, feed all the alerts into a single place, and then use data science and machine learning techniques to sift them and identify relationships with them. The result will be a shift in focus from individual alerts to the business problems those alerts are the symptoms of.

    Algorithms are great at these sorts of repetitive, high-volume tasks. They can look at massive volumes of events and figure out which are even relevant and worth taking a second look at, and then identify how those events relate to each other and what they really mean. This is where the humans come in, with their strengths in low-volume and unpredictable situations. Because they are not constantly drowning in irrelevant noise, they can work together effectively to understand the actual problem and get it fixed quickly.

    Where’s the Catch?

    The difficulty is that for many Ops people, there is a big cultural change required to adopt this model. The old deterministic systems may have required laborious maintenance, but at least it was possible to model them and understand why a specific result was achieved. New algorithmic techniques take a radically different approach, and while the results are impressive, they come from a black box. This does not inspire confidence for people who are very comfortable with the old approaches.

    Often, the response of Ops people coming from legacy tools is to get stuck into the minutiae of what is going on inside the black box and to look for traditional failure modes. The problem is that these approaches are no longer valid in the new world, as I discussed in my last post about getting to SRE.

    For instance, while it is indeed important to make sure that no issues are missed due to over-aggressive filtering, this does not mean that we need to see every single alert. As long as we know about the actual business problem, we don’t actually care about the single alerts, except insofar as they help us debug the underlying issue and resolve it.

    The other direction is even more insidious, where people spend a lot of time chasing down “extraneous” events that they think they should not be seeing. The risk is, of course, that the Ops team ends right back up in the same place that they were trying to escape, where they are only able to identify expected problems, but are getting blindsided on a regular basis by the “unknown unknowns,” the conditions that simply should not happen.

    Ultimately, Ops needs to let go of reflexes that were good in the old world, and focus on the goal: ensuring the uptime and stability of business-supporting systems. The how is less relevant than the what and the why; the algorithmic black box might as well be full of magic leprechauns, as long as they are telling Ops what they need to know, when they need to know it. Call it the “Chinese Room” theory of Ops: For as long as the results are useful to Ops, it does not matter what is inside the room.

    The benefit for Ops is precisely that they no longer need to focus on the minutiae of individual alerts, but can concentrate on solving actual business problems. That is how IT can make itself a value generator and avoid being viewed as another undifferentiated cost center.

    Pay no attention to the algorithm behind the curtain.

    — Dominic Wellington

  • The Time Is Now for an IT Operations Revolution

    The Time Is Now for an IT Operations Revolution

    There’s a level of frustration starting to build inside IT operations teams that’s about to boil over. Lucas Carlson, vice president of strategy for Automic, a unit of CA Technologies that provides an IT automation platform, says developers are bullying IT operations teams by programmatically invoking application programming interfaces (APIs) exposed by public cloud service providers. As the number of application workloads running in those clouds increases, IT operations professionals are losing control.

    To ensure their continued relevance, IT operations teams need to leverage automation to inject levels of agility into local data center environments, he says. That would make it much less likely developers would feel the need to programmatically invoke IT infrastructure resources.

    Carlson says achieving that level of agility will require IT operations teams to rely more on machine- and deep-learning algorithms to automate low-level tasks. Instead of thinking in terms of managing silos of compute, storage and networking resources, Carlson contends IT operations professionals need to reinvent themselves.

    The artificial intelligence (AI) models based on advanced algorithms must be trained continuously. IT operation professionals could become the trainers of those AI models. In fact, Carlson says IT operations teams are approaching a critical moment in history: They can either wait for AI models to eventually replace them or they can leverage them to take back control of IT infrastructure.

    However, developers tend to make decisions based on what they prefer versus what’s best for the organization. Before too long, multiple duplicate operations management frameworks are in place. Rather than incur all those management headaches and associated technical debt, Carlson says organizations need a single framework that can more adroitly accommodate the multiple roles within the DevOps team. It’s already long past the time when IT teams should have modernized their operations, he says.

    The relationship between developers and IT operations teams has always been contentious. The more belligerent members of the developer community are even making a case for “NoOps.” In that context, all the IT dollars spent on operations are moved into developing applications because the management of IT infrastructure is completely automated. Given the amount of legacy infrastructure that isn’t likely changing anytime soon, achieving that level of automation inside the enterprise won’t happen for a while. But there’s no doubt that IT operations needs to get more agile. Because of that issue, investment in software-defined data center (SDDC) technologies is growing rapidly.

    Instead of resisting that trend, Carlson says teams need to demand that internal IT operations be brought into the 21st century. It’s true that SDDCs infused with AI models will automate most of the lower-level tasks associated with DevOps. But those members of that IT operations teams who become the masters of building and training those AI models will be among the most highly sought-after individuals in IT.

    — Mike Vizard

  • Machine Learning Put to Work in the Digital Economy

    Machine Learning Put to Work in the Digital Economy

    IBM’s Watson already has a new retail job lined up to get a head start saving for its holiday spending this season. (Hey, even artificial intelligences appreciate buying gifts.) And Macy’s is hoping its customers appreciate its “Macy’s On Call” pilot, which combines Satisfi’s communications platform and IBM Watson cognitive technology for machine learning. Macy’s On Call creates an in-store companion on mobile devices that aims to help improve the service of customers. The pilot will be in 10 stores across the United States.

    According to the retailer, the Macy’s On Call app will be accessed via a mobile web browser. Customers can ask the app questions in natural language, such as where certain brands are located, and get customized answers. Customers also can learn about specific in-store services available at each test location, including personal shoppers, pickup counters or the location of restrooms. Macy’s hopes the app will also help them learn more about the services and features that shoppers want.

    Since each Macy’s store is different, and consumers increasingly prefer mobile platforms, the system will evolve as it learns more about each store’s customers. “At Macy’s, we remain focused on identifying, testing and supporting new ideas and approaches that will help elevate service to our customers through technology. With an eye toward innovation, we are moving fast to test and scale up pilot programs that help enhance their experience with us,” said Peter Sachse, Macy’s chief growth officer, in this statement.

    “This program, in partnership with Satisfi and leveraging the power of IBM Watson, will help us explore new ways to engage one-on-one with customers in-store, providing them another level of service right at their fingertips,” Sachse added.

    “Bringing Watson into a retail store setting presents an opportunity to engage with the consumer on a variety of levels,” said David Kenny, general manager of IBM Watson. “This particular use case takes Watson beyond helping consumers evaluate purchasing decisions, and influences another, equally important aspect of the in-store experience,” Kenny said.

    The first phase of Macy’s On Call recently launched and is expected to run through late fall. Of course, Macy’s has the potential to learn an extraordinary amount about its customers’ preferences through this program—preferences that otherwise wouldn’t have been so easy to learn without the aid of the Satisfi platform or Watson’s machine learning. Macy’s also may learn how to more rapidly improve its stores, manage inventory most effectively, and more quickly learn what is and isn’t working in its retail environments. That is, of course, if it can make Macy’s On Call engaging enough for customers to want to try, as well as provide enough value for them to want to use over time.

    Macy’s wasn’t the only one recently to make a significant machine learning announcement and the innovative use of technology. Google said it is using software from its DeepMind subsidiary to significantly cut the costs to power its data centers. According to DeepMind co-founder Demis Hassabis, two years ago Google began applying machine learning  to operate its data center more efficiently. And over the past few months, DeepMind researchers began working with Google’s data center team to significantly improve the system’s utility. “Using a system of neural networks trained on different operating scenarios and parameters within our data centers, we created a more efficient and adaptive framework to understand data center dynamics and optimize efficiency,” the Google team wrote in this blog.

    According to Google, the team cut power usage in their data centers by several percentage points, which translates into a 15 percent improvement in power usage efficiency. These are exactly the types emerging technology applications CIOs and chief digital officers need to focus on if they are going to succeed, as we covered in our post, “Digital Transformation Strains Traditional IT.”

    According to the McKinsey report, “Digital America: A Tale of the Haves and Have-Mores,” the United States is digitizing so rapidly that most are scrambling to adapt. The race to keep up with technology and put it to the most effective business use is producing digital “haves” and “have-mores”—and the large, persistent gap between them is becoming a decisive factor in competition across the economy, the report said.

    According to the report, “Digitization is happening unevenly, and users with advanced digital capabilities are capturing disproportionate benefits. The companies leading the charge are winning the battle for market share and growth; some are reshaping entire industries to their own advantage. But many businesses are struggling to evolve quickly enough. Workers in the most digitized industries enjoy wage growth that is twice the national average, while the majority of U.S. workers face stagnant incomes and uncertain prospects.”

    Clearly, organizations such as Macy’s and Google making moves want to ensure that in the year ahead they want to be the digital “have mores.” Businesses that aren’t making their own similar moves to employ machine learning to cut costs, improve customer experiences, or find other ways to advance the business may not find themselves the “haves” but the “have nots.”

  • Machine Learning, AI Driving DevOps Evolution

    Automation is the fuel that drives DevOps. Automating routine, repeatable tasks is one of the defining characteristics of DevOps culture. As artificial intelligence (AI) and machine learning improve, the scope and complexity of the tasks that can be automated increases, which raises the bar for all of DevOps. The humans behind the DevOps will be freed from even more mundane tasks and able to focus on more innovative and creative endeavors.

    It makes sense, if you think about it. The trend from physical services in an on-premise data center, to virtualization, to cloud services, to DevOps has been a steady march of raising the bar for automation. The more applications and entire infrastructures can be developed, deployed, monitored and managed programmatically and through automation, the more efficiently and effectively an organization can function.

    As tasks are offloaded to automation, humans can address other challenges. That focus inevitably leads to identifying other new tasks that can be automated. AI and machine learning represent a quantum leap in this evolution, however—with AI and machine learning, the automation can begin to identify and improve itself. When the automation can identify bottlenecks and inefficiencies and automatically adapt to overcome those obstacles, human input and interaction could become almost obsolete.

    In a recent interview, Christian Beedgen, CTO of Sumo Logic, shared some thoughts about the impact of machine learning on DevOps. “Machines will enable greater productivity among teams and businesses through the advances that machine learning and automation can offer,” he said. “A necessary component to augment human contributions, machine learning will finally give humans the power to review millions of bits of data.”

    Automating routine tasks is crucial, but there’s another factor that plays into the role that AI and machine learning have with the future of DevOps. The reality is that there are some things humans simply can’t do as well or as fast as machines—especially at scale.

    Steve Burton stressed in a blog post that humans are capable of observing, digesting and interpreting only so much information at one time. Some feel like the Holy Grail solution is some sort of application performance management (APM) dashboard that can aggregate and correlate real-time data so human IT personnel can monitor metrics and identify issues manually. “This all sounds good in theory, except it’s completely unmanageable in reality when your applications have thousands of components, billions of metrics and hundreds of changes every day,” he wrote. “Humans can no longer cope with this complexity, they now need machines to do the leg work, process this Big Data and provide operational insight into what the f**k is going on.”

    Beedgen agrees. He suggested, “Increasingly, businesses will adopt machine learning technology for its usefulness in augmenting human understanding of complex interaction and large data sets by uncovering the unknown unknowns.”

    On some level this all sounds a bit scary. It seems as though the more we succeed at DevOps and the more DevOps evolves, the less humans are even a necessary part of the equation. Conceivably you could be so good at DevOps that you DevOps your way right out of a job or inadvertently launch SkyNet—or both.

    That scenario is simultaneously feasible and far-fetched. I suggest that you don’t fear that future or look at it as machines stealing your job or taking over humanity. Embrace the future of DevOps driven by AI and machine learning and find the silver lining. Seek out the new frontiers and horizons, because there always will be tasks that the machines simply can’t do without us—and that is where innovation happens.

  • Avik Partner’s Grok APM Uses Machine Learning to Find Anomalies Early On

    Avik Partner’s Grok APM Uses Machine Learning to Find Anomalies Early On

    Machine Intelligence-Driven Anomaly Detection

    Machine learning—the science of getting computers to take thoughtful action without explicit programming—is the basis of Machine Intelligence-Driven Anomaly Detection, which is the core capability in Avik Partner’s Grok APM product, according to Casey Kindiger, veteran enterprise software solutions provider, IT automation expert, and founder of Avik Partners.

    “Thoughtful action means using the type of faculties that a human being would use to make a decision and to take an action,” says Kindiger, who has more than a decade of experience in designing and developing process automation solutions for companies such as JP Morgan Chase, T-Mobile, and Allstate Insurance. Grok learns from your dev and ops environments and takes thoughtful action to detect anomalies, all in the service of Application Performance Management.

    Grok works by analyzing real-time performance data from apps and infrastructure, looking for behavioral bumps in an otherwise smoothly-paved DevOps highway of deployed services and underlying infrastructure.

    Grok’s Pre-Event Inferences

    “Grok uses common statistical techniques to learn a pattern over a selected period of time in order to build a dynamic threshold of the continuously changing underlying patterns,” says Kindiger. In this way, Grok can establish a sort of dynamic baseline from very noisy patterns, which is something that would typically take a number of person hours of involved statistical and predictive analytics to accomplish.

    Grok’s machine intelligence leverages the ability to make inferences based on this dynamic baseline information like a human being would in order to reason out a logical conclusion. Grok does this to detect the slightest of anomalies based on the difference between what reasonably should have happened and what actually did happen. Using this approach, Grok detects subtle patterns of significance in streaming data despite the fact that the data has patterns that are continuously in flux.

    But rather than detect when something has gone wrong at a point in time, which is not scalable or sustainable, Grok looks for minute patterns that appear well in advance of any issue that you would normally consider looking at with traditional APM mechanisms, explains Kindiger. Rather than send alerts based on threshold-based exceptions, Grok makes a prediction on what the next data point or the next series of data points will be and then looks at deviations from those predictions, considering whether to flag those as anomalies, according to Kindiger.

    Currently, Grok is particularly tuned to analyze streaming data and is, according to Kindiger, data-point agnostic. “We don’t need to teach the system what the context of a particular data point is. The system learns the context of the data over time and the subtle nuances of the pattern that produces over time, and creates the model based on that,” says Kindiger.

    Watch Out for That Initial Two Week Delay

    If Grok does everything that Avik Partners’ Kindiger hypes that it will, DevOps teams will soon be able to catch and act on infrastructure issues before they grow big enough for end-users to notice.

    But Grok does have one wrinkle in its APM armor: it takes time for it to really learn a new customer system and get good at detection. Let’s assume that Grok’s machine learning causes it to get better at detecting anomalies in a given set of infrastructure over time. That means that there’s a point, at the beginning, given a brand new customer when Grok only starts to learn. At that point, Grok isn’t very good at knowing what’s anomalous compared to what’s normal for that infrastructure because Grok has no foundation for that particular system.

    “Grok will take some time to learn the nuances of when to escalate an issue. We’ve learned from experimentation that it gets pretty good after ingesting two weeks’ worth of data,” says Kindiger. Because Avik Partners initially targeted AWS, it has taken two weeks of CloudWatch data and fed that to Grok to provide it with some backdrop so that when Grok goes live, it won’t be learning this environment entirely anew.

  • How machine learning could head off security quality failure

    How machine learning could head off security quality failure

    With so many players in the mix from operations to developers, from PMs to technical leads, to Q&A, most of their activities are automated. Systems record all their email, Wiki, and chat conversations and other notations about these activities. This gives the devops community the opportunity to learn from that data in order to improve future projects.

    “We have access to a massive amount of data that can in total define what the process is that’s going on and define it in-flight in real-time,” says Aaron Cois, Researcher, CERT Division, Software Engineering Institute, Carnegie Mellon University. Through exploratory research, Cois is learning that by using machine learning tools, enterprises can analyze security activities in the development process based on data from issue trackers and commit logs.

    Valleys in Topic Density Point to Software Development That is Missing the Mark

    Natural Language Processing, a subset of machine learning, uses systems to process text in unstructured data and to perform topic detection, determining whether or not a portion of the text belongs to a given topic category. “In our work, we’ve taken text from issue trackers and used machine learning to classify each issue as security-related or not security-related,” says Cois.

    This has practical applications for detecting the course of a project and steering it in the right direction. If the PM expects the developers to raise the quality of security for an iteration of the software package, he can monitor the chatter among developers about the issues they are adding to the issue tracker or he can monitor the comments on the code.

    “By using machine learning to classify issue tracking or source code commit messages as security-related, we can see a reasonable measure of the amount of work that’s being done to ensure the security of the software system,” says Cois.

    If the PM can determine based on a shortage of security chatter that in actuality not much security work is going on, he can ask why, what is the challenge? “Is the security stuff more difficult? Did we do our due diligence to allow our developers to adequately address security?” asks Cois. Then the PM and the teams can work together to remove any roadblocks before any more scope creep or veering off course occurs.

    Machine Learning for Physical Modeling and Trends Analysis

    If the industry can build statistical models of what a healthy software project looks like as it progresses, then it should be able to determine whether a given project is on track and what risks it may face along the way.

    Issue tracking data, commit logs, wikis, chat server logs, build/deployment logs—every type of data that can lead to a full picture of project activities—would form the basis of these models, according to Cois.

    Through Natural Language Processing, machine learning can help determine from this data what the rates of creation and resolution of issues are and what the rates of commits and overall progress are for the software project.

    Machine learning can help PMs look at how their project’s rates in these areas compare with rates of similar projects, which developed certain problems or turned out in certain ways. If there is a strong correlation between rates of projects that had known outcomes and rates in your project, then you might want to use that information to make corrections early on, in cases where the outcomes of the other projects were undesirable.

    Over time, the models become stronger and their predictive value increases, leading to some strong correlations between current projects and the models collected. With the increasing density of the data available in models as they grow, even the weakest numbers in early models can become stronger, giving the model strong correlative value in areas where strong correlations did not appear at first.

    “If I have collected both activity and outcome data on thousands of projects, and a large number of them are highly similar to the current project, I’ll have much more confidence in my predictions about the potential outcomes of the current project given the behaviors I am observing,” says Cois.

  • From Orwellian Nightmares To Unicorns: Why I’m All In On DevOps

    From Orwellian Nightmares To Unicorns: Why I’m All In On DevOps

    Over the years I’ve helped literally thousands of customers gain efficiencies in their data centers. I’ve met and become friendly with a lot of IT pros but I’ve got to admit – I never envied them. Well, maybe I envied some of the views in NY and Jersey City. And some of the toys they got to play with over the years were amazing.

    Nonetheless, I didn’t envy them because many of my friends in director and VP roles spent a huge percentage of their time fighting politics that resulted in Orwellian distortions of the truth. A couple of stories from the early 2000s might help illustrate my point:

    1. Two guys pulling all-nighters delivering a $1.5B roll out of IP communications while 25+ project managers debate progress from 9-5

    The headline pretty much captures it. This now too-big-to-fail bank, as well as their vendor, had brought in several project and program managers to ensure groups from facilities through supply chain through security and operations were “aligned.” I spent a couple of meetings getting to know all of these project managers before realizing that there were two guys working every night, all night, to manually, using fingers and spreadsheets, push and validate changes. Eventually those operators quit – and the project ground to a halt.

    All the alignment, of course, was pretty much BS. At one point I mentioned in a meeting that our software would not run on the same server as the IP Communications System (Call Manager for those curious). I was corrected – “it isn’t a server.”

    I responded with something like, “it is an HP DL something or other. Isn’t that a server?“

    Answer: “No, if we call it a server then the server team will need to get involved and we don’t want the server team getting close to this project.”

    You cannot make this stuff up. Servers = / servers. Got it?

    2. Similar story, different bank – 1% of achievement of goal = “on plan” to outsourcer

    The story starts when I was brought in by a huge vendor to a bank that is also too-big-to-fail. This bank also had a high profile project to get the bank over to IP communications. Keep in mind that this was after 9/11 and all the banks were looking up to the teams at Lehman that had been able to get Lehman back trading significantly faster in part because of their use of IP communications. So the project to get the bank onto a new approach to real time communications had CIO visibility.

    One of the most surreal meetings of my career occurred when after having been vetted, tested, recommended, and designed into the solution I was granted an audience with the SVP handling the bank at EDS; I feel ok with naming EDS since they no longer exist and I’m sure this executive has retired. Like many folks at EDS he was former military and as an account executive he owned a north of $1B outsourcing contract.

    I started off by saying that we can help you get back to schedule, as I know you have only done about 1% of the deployments originally planned. He looked me right in the eyes and said: “EDS is on plan, and we always will be.”

    After some back and forth I got it – no matter what the facts on the ground were, he was never going to accept any culpability at all for the project delay. So missing the project time line by 99% was officially “on plan.” That’s Orwellian.

    DevOps restores faith in humanity / IT

    Compare any decent “DevOps” shop to the above stories and you can see why I’m so enthusiastic about doing my small part to help bring DevOps approaches to operations to a broader market.

    meme-damn-good

    The teamwork between not just Dev and Ops, but between everyone responsible for delivering capabilities to the organization, is so refreshing. Just last week, Google engineering blogged about using machine learning to improve operational efficiency of their data centers, with a focus on power consumption and so forth. The technology I get – we are headed down a similar path with deep learning and control theory. But what struck me more than the technology was the data they are using to draw inferences would often be siloed and tightly controlled by groups within IT at legacy companies precisely because it could be used to score points.

    So DevOps entails more than infrastructure and code and collaboration. With DevOps, IT organizations move from fighting over terms like “server” and “on plan” that are often filled with well meaning and intensely frustrated teams towards an environment where cool stuff happens all the time. And IT done right – making sense of the world with software and hardware – is incredibly cool.

    I’m betting on DevOps because I want the world to be filled with DevOps unicorns and because, like my friends that are operators within legacy environments, I cannot stomach more Orwellian obfuscation of the facts on the ground for political expediency. DevOps makes the world smarter and IT teams a heck of a lot happier. What’s not to like?