devopsy.kinsta.cloud

Tag: IT operations

  • AIOps: The Path to Greater Resilience and Uptime

    AIOps: The Path to Greater Resilience and Uptime

    Large organizations with complex IT infrastructures are facing a significant challenge when it comes to data. The complexity is such that many IT operations teams are struggling to detect, or predict, the complex failure conditions and circumstances that can adversely impact resilience and uptime.

    Monitoring a multitude of systems is a complex task, as the data generated by any system can be vast and far too overwhelming for manual IT operations processes, particularly in relation to analyzing, correlating and prioritizing alerts and remediation.

    Companies have much to gain from a better understanding of the data being created from monitoring tools and practices: increasing the resilience of systems, and the speed of remediation cycles delivers significant cost and time savings. Harnessing advanced technologies to deliver more intelligence to teams in charge of performance monitoring is becoming increasingly important, which is where AIOps can deliver significant value on top of existing monitoring and automation tools.

    Cutting Through the Noise

    Telemetry tools deliver staggering amounts of data. The Microsoft Azure telemetry platform, for example, records 10 petabytes of data per day. When dealing with this volume, the ability to intelligently decipher the severity of alerts becomes vital, especially as many of them may be duplicates or not meaningful in any way. Yet it is IT operations teams that are tasked with understanding how systems are performing, and where resources should be diverted to ensure services are always up and running. To do this effectively, the root causes of alerts must be identified quickly and correctly.

    AIOps is increasingly becoming the go-to solution to the problems inherent in dealing with big data. By applying data collection, data modeling and data analytics techniques and using machine learning algorithms to establish patterns, the noise of the data is reduced as operations teams gain more intelligent insights into alerts. So, providing more context to issues flagged within systems is the goal. Eventually, somebody has to take action when an alert is created, so the more intelligence engineers have, the easier it will be to remediate.

    The less context delivered with alerts, the more time engineers will need to resolve the issue, as the manual remediation process will likely involve interaction with other teams. Take application performance monitoring as an example. In my experience, operations teams do not have an end-to-end understanding of the applications that are live in the production environment. If an alert is created, the ops team will typically seek assistance from a member of the developer team who built it, or another colleague with a deeper understanding of the application, which of course extends the lead time of the remediation process.

    Key Considerations for Deploying AIOps

    By establishing data patterns through machine learning and other advanced analytics technologies, the following key drivers for alerts assessment using AIOps can be established:

    Frequency: Discovering how regularly alerts are created and whether it warrants automation for analysis, correlation, prioritization and remediation.

    Coverage: Telemetry is of course vital to AIOps, as the machine learning algorithms will only be as good as the data they ingest. Putting in place the right telemetry tools is therefore vital to ensure the right data is being gathered. Failing to do so increases the risk of missing critical issues.

    Impact: Not all problems require equal energy and resources. Certain issues might only occur every few months and may not be critical. In such situations, it is not prudent to invest significant amounts of time and money into developing the machine learning algorithms that will identify where and when an alert will be created. Again, cutting through the noise of the telemetry will help to identify high-priority alerts.

    Probability: What is the likelihood of certain issues recurring? Can automation tools, informed by the machine learning algorithms, be implemented to deal with these, i.e. learn where and when they are likely to occur so they can be dealt with by the AIOps platform autonomously?

    Take, for example, the issue of CPU or memory going beyond a certain threshold. Rather than manually contacting customers when an issue is flagged to inform them that services are about to go down before being spun up again, an AIOps platform can automate this process, so remediation happens seamlessly.

    Latency: Automation of the remediation process for an alert deemed critical requires increased investment in AIOps. Getting good data from the early stages of AIOps processes will reduce the amount of time taken to both flag and deal with a problem. Containerized microservices, for example, take very little time to recover, so very little investment is needed when applying AIOps tools that automate remediation. Database recovery, however, is a more complex process, the automation of which would require significant investment.

    What’s the Customer Value?

    For service providers, AIOps is essential if they are to attain the highest levels of resilience, the measure of which all comes down to the Service Level Agreement (SLA). The SLA is measured by a percentage that corresponds to the time systems and are up and running over a year. So, 99.99% uptime is good, 99.999% is great, but 99.9999% is where service providers want to be. To put this into context, downtime for an entire year in the latter scenario amounts to just 32 seconds.

    Increased automation is the only way to achieve optimal SLA. To do this, service providers are increasingly turning to AIOps solutions.

    Operations teams are in charge of the IT strategies that align with business objectives. For service providers, a key part of this is, of course, ensuring maximum availability and optimal performance of the customer-facing environment. Understanding when and where faults appear, and improving resilience against them, is the best way to ensure this. That is where AIOps truly adds value.

    — Prashant Jain

  • The Role of Containers in Business Transformation

    The Role of Containers in Business Transformation

    Containers hold new promise for business and digital transformation. Yet many differences exist between monitoring containers and monitoring other elements of infrastructure, such as VMs, storage, memory and compute. Understanding those differences is vital for ITOps managers as they navigate their businesses’ digital transformation.

    Interestingly, the differences are as much technical as they are organizational. Companies need to explore these differences and consider questions such as, how to ensure that a container-based project performs when it goes into production, and what changes you must make to infrastructure to ensure you have enough compute and storage power to support the containerized applications your development team is going to launch.

    The essence of container management is to be prepared for the changes coming from application development, which means understanding resource and infrastructure as it is today to estimate the effect of those changes tomorrow.

    Here are four differences ITOps managers need to consider with container management as they move their digital transformation journeys forward.

    Containers Versus VMs

    The signature feature of containers is they can hold and run an entire application all in isolation. VMs offer operating system level isolation, but at the cost of a much larger footprint; to begin with, each one has its own copy of the entire operating system. VMs are also far less portable than containers, especially when moving among private and public clouds and data centers. As a step toward digital business transformation, most companies are looking for DevOps tools and for ways to deploy their applications to the cloud, whether for QA or production. Organizations rolling out new applications are also gravitating to a container-based architecture from the start.

    Advantages of adopting containers include:

    • Agility: Instantiating a new container goes as much as 20 times faster than instantiating a new VM. Startup is measured in seconds rather than minutes and once deployed, containers scale up and down rapidly in response to load.
    • Density: Most containers are megabytes in size rather than gigabytes. With their smaller footprint, containers allow for as much as five times greater density than VMs.
    • Cloud-readiness: Containers can be deployed to any cloud (private, public, virtual or physical) which opens up the possibility of using the same containerized application in hybrid and multi-cloud environments.
    • Blast radius: As software developers incorporate and release changes more quickly, they look for ways to control potential damage from bugs. A problem in a container is less likely to cause widespread damage and is easier to isolate and repair than a problem in an entire VM or physical server.
    • Cost savings: Every VM requires a hypervisor and an operating system (OS), which can involve licensing and fees. Containers run side by side, sharing the OS kernel of a single physical machine while holding isolated copies of specific library versions they depend on. Architecturally, VMs tend to contain more components and require greater allocation of resources. Containers, on the other hand, tend to hold a specific component or a single micro-service, minimizing resource overhead.

    Containers, DevOps and Real-time Monitoring

    A company adopting DevOps to speed up deployment of its software products automates as many development processes as possible and relies heavily on automation scripts to move code along, from continuous integration through continuous deployment and continuous delivery (CI/CD/CD) to production. Along the way, multiple developers must check in on new changes. Automated processes, such as health checks and unit testing, take place and then the build is deployed to the next environment, such as integration or QA. With each set of changes, ITOps needs the answers to several questions:

    • How will this affect performance?
    • What will be the impact of the changes when we deploy?
    • Do we have enough compute, storage, memory and networking to support the changes?

    In a traditional environment, developers have plenty of time to answer those questions. But if they’re deploying modern, containerized, cloud-based applications, that window tends to be much smaller which can lead to the kind of guesswork that makes ITOps uneasy. That’s where container monitoring comes in. When ITOps can see real-time and historical analytics of containers and their hosts, across physical, virtual and cloud environments, they can create performance benchmarks to help them make informed choices about infrastructure.

    Microservices Get Added into the Mix

    Microservices are loosely coupled and independent services, each with their own release schedule and lifecycle. Developers are able to make small changes to elements of these applications which allows them to mitigate the risk of negative business impact by reducing the blast radius around those changes. In case of failure in a single change, there’s less to repair and less impact on surrounding, dependent services. We’re seeing developers move away from packaging a single application per container to packaging a single service of an application per container. This allows them to make changes more frequently and keep up with business demands.

    Multiple Silos

    The goal of DevOps is to bring application development and traditional ITOps closer together. However, container monitoring and management changes things. Traditionally, areas such as storage, network bandwidth, VMs and physical hosts that require monitoring have all been in the ITOps silo, an area that developers don’t typically own. Developers write code and build software in the application development silo and then hand the software off to the ITOps silo to run. However, things have changed and now developers aren’t only writing code, but also owning and using containers in the application development silo. Once the software and containers leave this silo, their consumption of resources like compute, storage, network bandwidth and memory will also affect the ITOps silo. That’s why monitoring and management is as necessary for containers as it is in areas like storage, VMs and physical hosts.

    While container technology may be new for the traditional enterprise, the impact it will have on business transformation cannot be ignored. Containerization brings increased simplicity, consistency and portability to production environments, enabling faster IT delivery and increased performance, arming businesses with a new agility and allowing them to rapidly respond to changing customer needs. In fact, according to 451 Research, application containers will be a $2.7bn market by 2020.

    The businesses that embrace this opportunity will be taking a vital step towards transforming into bustling digital enterprises that can deliver innovative products and top-notch customer experiences. To learn more about containerized infrastructure and cloud native technologies, consider coming to KubeCon + CloudNativeCon NA, November 18-21 in San Diego.

    — Yinghua Qin

  • What is so Special About AIOps for Mission Critical Workloads?

    What is so Special About AIOps for Mission Critical Workloads?

    As the wave of digitization sweeps more enterprises with each passing day, the threats posed to organizations, particularly IT organizations, have never been more complex and pressing. With cybercrime statistics, including those for more targeted crimes (such as phishing) pointing towards an alarming frequency of attacks, the stakes for securing enterprises have been higher.

    However, many IT organizations have been unable to smoothly transition into pushing the digitization of their organizations forward. This inability to support and drive digital transformation isn’t a result of a lack of effort on behalf of these organizations. Instead, it has everything to do with the unavailability of the proper management tools needed.

    Unfortunately, many IT organizations lack the tactics and tools needed to overlook the smooth shift to digitization on such a grand scale, which often leads to the degradation of critical business applications, along with IT slowdowns and outages. Luckily, however, artificial intelligence for IT operations (AIOps) arrives like a beacon of hope for revolutionizing IT application, along with increasing business agility.

    The Failure of Enterprise in Tackling the Burgeoning Complexity of IT Infrastructure

    A primary reason behind the ever-present issue of IT slowdowns in enterprises and organizations that is frequently overlooked is the fact that the customer-facing side of IT (applications) is growing increasingly complex with each passing day.

    As the increasing number of applications continue to occupy more and more space, the IT infrastructure grows into a vast and hybrid structure which often spans across multiple clouds on a grand scale.

    Although the vastness of the IT infrastructure is a byproduct of the digitization drive taking enterprises by storm, it is still worlds away from the security infrastructure of yesterday, which was much easier to manage and analyze.

    From the Internet of Things (IoT) connecting thousands of devices onto an organization’s network to complex issues such as bottlenecks, application slowdowns and outages, along with performance latency and the ever-present threat of phishing attacks via a free host website (000webhostapp, being the most commonly used host), organizations are just setting their systems up for failure at some time in the foreseeable future.

    Owing to the flourishing complexity of the IT infrastructure, an organization’s system fails to cope with new, dynamic workloads for any multitude of reasons. Usually, the system is utilizing extra resources or is under the influence of a recent addition to the system.

    In instances of outages or IT slowdowns, the pressure to fix the IT infrastructure falls on the IT team, who are tasked with the immense responsibility of analyzing and diagnosing the issues as they occur in real-time, along with coming up with a solution that satisfies customers and prioritizes the business aspect of an organization as well.

    Unfortunately, however, without access to the proper tools and techniques, the IT team can only conjure security patches that resolve the problem for the moment but fail to withhold security in the long run. Also, without the ability to effectively predict a pattern for the ever-present security threats, the IT team is operating on a whim rather than basing their solutions on the rationale.

    How Can AIOps Help the Situation?

    To leave the momentous challenge of perfecting an organization’s IT infrastructure solely on the IT team is not only outdated, but it is also impractical. Taking into consideration the complexity of the threats facing enterprises, a solution that arches above each aspect of cybersecurity in an organization should be implemented.

    Today, that solution arises in the shape of AIOps. The hybrid IT infrastructure has become too complex to be handled through traditional tools and tactics, which is why AI, along with machine learning and automation, has to step in to bridge the gap.

    Unlike other security strategies, AIOps integrates a wholly modern and unique approach, which allows for the AI to view systems in terms of business outcomes. Simply put, rather than analyzing and diagnosing identities, infrastructure and applications independently, the AIOps platform follows the path taken by the user into the complex IT infrastructure and back out of it.

    Ideally, AIOps platforms run automated investigations that derive the root cause of existing and recent issues on a regular basis. When it comes to managing a business-critical application, an AIOps platform can detect an unusual response within the system, whether it’s an anomaly in a workload profile or variant response to critical workload.

    Other benefits to incorporating AIOps as a security tool include tapping into the truly unique set of capabilities offered, including predictive capacity management and forecasting/predictive abilities. The predictive ability of AIOps allows for the AI to smartly notify IT teams about any attack or potential outage, or slow down, before they actually happen so they may act accordingly.

    By taking a different approach to the pressing issues faced by enterprises today, AIOps manages to do what several security strategies have failed to accomplish—provide organizations with an end-to-end, app-centric approach to security, which also takes into consideration the context at hand.

    Moreover, AIOps will push organizations to shift their focus from reactive problem solving and cause them to optimize their IT infrastructure on a more proactive basis. Fundamentally, AIOps provides IT teams with the necessary tools to diagnose and combat both internal and external threats to an organization’s security infrastructure.

    After the integration of AIOps within an organization’s infrastructure, enterprises can turn their attention into optimizing the balance of critical workloads, along with prioritizing business functions and improving applications, rather than worrying about security.

    What Hurdles Can Enterprises Expect with AIOps?

    Despite the countless benefits AIOps brings to the table, certain obstacles arise when it comes to the implementation of the security model into organizations.

    Perhaps the most apparent obstruction organizations have to deal with is the extensive amount of time investment AIOps requires. Like every other security tool centered around an AI or machine learning technology, time is absolutely pivotal for the learning to actually take place. A simple way to smoothen the transition, AIOps must be fed with data, statistics and analytics originating from multiple data centers.

    Another common issue that arises, mainly because of a lack of research, is when an AIOps platform fails to perform as per the specific requirements of an enterprise. Keeping this in mind, CIOs must voice their concerns about fine-tuning AIOps before committing to a vendor.

    AIOps abilities in performing an in-depth analysis into the IT and security infrastructure—within the context of the applications it performs—gives the unique chance to stay ahead of the threats they face.

    — Rebecca James

  • Chaos Engineering for ITOps

    Chaos Engineering for ITOps

    Chaos engineering (CE) is the discipline of experimenting on a system in order to build confidence in the system’s capability to withstand turbulent conditions in production. This approach is becoming commonplace in software development and operations (DevOps) practices. But how would its application extend to ITOps? CE for ITOps offers a similar framework for stress-testing a technology platform to understand its weak points and performance pitfalls under heavy pressure.

    CE tends to be used primarily in DevOps during bug testing: setting up experiments to run software under different conditions, such as peak traffic, and monitoring how it functions and performs. This becomes increasingly necessary in cloud-based systems where failure to understand extreme load responses could result in runaway cascade failures or, worse yet, spinning up thousands of extra nodes handling error conditions while not doing any actual work. These same principles, applied to IT operations management (ITOM), help define a functional baseline plus tolerances for infrastructure, policies and processes by clarifying both steady-state and chaotic outputs when extremes are reached.

    Applications in IT

    The theory of CE in DevOps gained early traction at Netflix as they moved from physical to virtual infrastructure, with the team that implemented it on AWS breaking off to form Gremlin. However, CE is not typically used in ITOps, because ITOM has historically been separated from development (generally, IT monitors system dynamics, and when a problem occurs, engineering change management or ITSM is brought in to remediate the issue).

    With the growth of containerization in cloud applications today, IT infrastructure looks more like development environments than classical multi-tier architectures. But the limitless scale of the cloud means failures can also be limitless: Microservices are well-served by testing elasticity and scalability, data flows and resiliency through stressing the system to the edge of its tolerances and fixing their shortcomings before a public crash.

    One, Two, Three…Chaos

    Implementing chaos engineering for ITOM provides a systematic approach to identifying weaknesses in a microservices-world. In a monolithic environment, you have visibility into performance and event metrics that may be lost with microservices designs. As a result, the need for operational insights becomes even more critical when scaling to unknown workloads.

    Netflix’s Chaos Monkey grew out of CE principles from their own cloud-native community, meant to address the gaps in common dev tools’ abilities to manage extreme complexities. This methodology is extendable to infrastructure and helps to set guardrails on platform behavior as a whole.

    Here are five fundamental steps to follow in order to bring this thinking into your team’s ITOM.

    Define the Current Steady State

    Performing baseline analysis is a standard concept in capacity planning, upgrade strategies and other high-impact functions. Start with something relatively simple (and small) so you don’t get overwhelmed by the data, or risk interfering with the business if something goes wrong (such as security Red Teaming). For example, monitoring CPU and network utilization, which are common bottlenecks in any IT shop.

    Define Optimal Conditions

    There’s how your system generally operates, and then there’s how it should operate; these typically aren’t the same thing. CPU utilization and network latency are always affected by application efficiencies, hardware conditions and a host of other factors. Create a standard that outlines what engineers should expect on a normal day, on an easy day and on a very hard day. These are the control groups, and the extreme day will be the stress test.

    Form a Hypothesis

    Where will the system break? If you’re running an application scenario such as doubling the peak traffic that even your worst day so far has seen, will your CPU maintain optimum utilization (or will the container provisioning engine smoothly deploy additional nodes) as in the variable control groups, or will it spike so severely that processes grind to a halt because there isn’t enough memory or network bandwidth left to manage the load?

    Execute a Real-World Event (But Contain the Blast Radius)

    Do something extreme, such as taking down a firewall that severs connectivity to one internet service provider. This will confuse the application as it tries to respond to requests with repeated failures, ramping up CPU processes as errors return from a dead network endpoint. Log events will mount, filling the database and saturating the backbone.

    Validate the Hypothesis

    What happened? Monitor utilization and network throughput during the test and see where the system fell over. Is it what you expected, or did something never previously considered take place? Did new chaos erupt from the fissures in your infrastructure? Stabilize, document and remediate.

    Never Stop Not Being Afraid

    Stressing a system to its absolute max—and a little bit further—to see where things go wrong allows you to understand steady-state behavior and error-handling, so you can fix it before something breaks in new and unexpected ways. What do traffic spikes look like? What are real-world events and their impacts on your organization?

    CE is not just for DevOps. It should be a systemic practice for load-testing (out of your comfort zone) to the point of failure. It’s a responsibility for more than microservices deployments and applies to all sorts of disciplines within the IT organization.

    — Michael Fisher

  • GitLab Outlines Scope of DevOps Ambitions

    GitLab Outlines Scope of DevOps Ambitions

    At the GitLab Commit event in New York today, GitLab launched an ambitious effort to extend the scope of its DevOps platform to provide a view into the software development life cycle for not just developers but also designers, product managers and business leaders.

    Fresh off raising an additional $268 million in funding, GitLab now plans to create different ways to view a common pool of metadata and metrics generated by the GitLab DevOps platform.

    It may be a while before GitLab can deliver on that promise, but the core concept builds on the dashboards the company already has built for IT operations and cybersecurity teams. Rather than requiring each team participating in the software development process to acquire a separate platform, GitLab should be viewed as a DevOps platform delivered as a single application that can now be applied to everyone, said Mark Pundsack, vice president of product strategy for GitLab, in speaking with conference attendees.

    As part of that commitment, Pundsack noted GitLab is expanding its DevSecOps capabilities to include auto remediation. The scanning capabilities built into GitLab will be extended not only to detect issues based on the cybersecurity policies of an organization, but also, when allowed, automatically remediate routine issues, Pundsack said, adding GitLab also will automatically roll back applications if necessary. Once in place, GitLab will have pushed application cybersecurity much further left than anyone might have expected when DevSecOps was first broached as a concept, he said.

    At the conference, GitLab demonstrated a pipeline graph capability that will provide a foundation for many of these promised offerings. GitLab already can be employed to accomplish tasks such as code quality scanning, container scanning, dependency scanning and licensing dependencies.

    All these features are an extension of GitLab’s existing strategy that aims to rationalize the number of tools organizations need to stitch together to drive a set of best DevOps processes. However, other DevOps platform providers have similar ambitions.

    Thanks to its latest round of funding, GitLab—which also announced its plans to go public next year—is now valued at $2.75 billion. The company has not released any revenue results but does claim to support more than 100,000 organizations using its platform, including Ask Media Group, Charter Communications, Delta Air Lines, Goldman Sachs, Ticketmaster and Nvidia. GitLab also claims to have more than 4,800 active contributors to its open core platform. The company now has more than 800 employees in more than 55 countries across the globe, and there currently are 222 open GitLab jobs posted.

    It remains to be seen how much $268 million will enable GitLab to fend off rivals such as GitHub, which was acquired by Microsoft, and CloudBees, which has built a large DevOps community around the open source Jenkins continuous integration/continuous deployment (CI/CD) platform. Whatever the outcome of this arms race, the massive amount of funding pouring into the platforms soon should take DevOps platforms to places many might not have ever imagined possible.

    — Mike Vizard

  • A Culture-Driven Approach to DevOps Transformation

    A Culture-Driven Approach to DevOps Transformation

    One of the most challenging aspects of a DevOps transformation isn’t automating and overhauling technology. The biggest challenge of DevOps is changing an organization’s culture and purpose. You can’t “buy a DevOps.” DevOps demands the involvement of people, process and tools working together.

    Unfortunately, I’ve seen too many companies get carried away with the latest shiny object and neglect the people aspect. So, the end results they were expecting do not come to fruition. Research from Google Cloud and Harvard Business Review supports the importance of a people-first DevOps strategy. This research confirms DevOps transformation projects are worthless without addressing the people aspect.

    Of course, such large-scale transitions are difficult. People have to put their reputations on the line to sell the transformation. Employees may become stressed or resist the cultural changes that come with a DevOps transformation. Therefore, communication throughout the entire transformation must be your primary goal. Communication includes measuring the effectiveness of the transformation vision and goals and sharing that information with your team. Keeping them informed ensures they understand the critical role they play in this vision, as well as the value of their work. These measurements need to also show how participation in change translates to value for the business.

    Here are some best practices for prioritizing people and process when working through your DevOps transformation.

    Communicate Every Step of the Way

    It’s important to establish a level of empathy for your employees and understand exactly how changes will affect them. Employees’ professional identities develop over many years, and you can’t expect them to change overnight. Here are some questions your employees are wondering about:

    • What happens to my job when we have to start adhering to this new process?
    • Will I lose my job when all this automation is implemented?
    • How am I going to learn the skills to be effective and contribute to this new way of working?

    Expect their questions and listen for more areas of concern. Be proactive about sharing both the questions and the answers with everyone.

    It’s critical that you keep an open channel of communication with your employees to help them through this change. Don’t view this as an add-on exercise–it should be the cornerstone. Focus on communication, and devote time to your staff, so they are completely integrated into the new DevOps process.

    Your communication strategy needs to include all the teams involved with application delivery. Hold regular meetings across all departments, not only development and operations teams. Think about how the DevOps transformation impacts departments such as marketing, human resources and finance. What challenges have they run into? Have there been any negative impacts thus far? Listening and addressing each issue in real time will ensure the transformation is moving in the right direction.

    Measure What Matters

    When it comes to measuring your success, don’t worry about what other companies may be doing. The metrics that matter are the ones that support your ultimate business goal, which in this case is a better software delivery process.

    Identifying metrics for your business demands you understand how your company’s process drives behavior. Ask questions such as: What corners were cut to get from development to production on time? Did meeting this deadline impact the timelines and performance of other teams and the quality of other applications?

    Meaningful metrics allow you better insight into your processes in the real world. And meaningful metrics help determine whether your pipeline is moving in the right direction to deliver high-quality, high-value software.

    Remember those regular meetings I mentioned earlier? These are also a great time to focus on metrics so employees understand how their work impacts the business. It’s also an opportunity to hear what’s happening on the ground in the delivery pipeline. This real-world information can determine progress, as well as test if the right metrics are being tracked.

    Conclusion

    A DevOps transformation requires changes to staffing, organizational structure, performance and culture. As a result, the success of your transformation involves moving the entire organization forward—building a strong culture ensures all people at all levels are invested in it. Measuring and communicating results must be the standard for every release you do.

    DevOps can fundamentally change any organization for the good. By continually communicating a clear vision of the goals and expectations of your transformation, people will feel confident they are on a winning track.

    — Tj Randall

  • DevOps Deeper Dive: BMC Software Bets on AIOps Transformation

    DevOps Deeper Dive: BMC Software Bets on AIOps Transformation

    As BMC tests the AIOps waters, the company seeks to find a balance in serving both the DevOps and ITIL populations

    It’s clear in the enterprise IT space that artificial intelligence (AI) is poised to play a significant role in IT operations, a phenomenon known as AIOps. Often with this level of IT disruption, new entrants funded by venture capitalists flood the segment, leaving incumbents to either fend off the new rivals, cede ground or, in many cases, put an end to the threat by acquiring one or more those startup companies that pose a threat to their existence.

    In the case of BMC Software, it has decided to stand its ground by aggressively investing in AIOps research and development. Most recently, BMC extended the AIOps capabilities of its existing portfolio of TrueSight IT management software through additional advanced analytics to identify patterns and anomalies within an IT environment. Other examples of BMC employing machine learning algorithms to automate IT environments tools include the ability to surface continuous recommendations for cost optimizations and automated event remediation workflows that promise to reduce the mean time to repair (MTTR).

    BMC has been making a case for AIOps as far back as 2017, but it’s only now that many organizations have enough confidence in AI to start embracing AIOps beyond the pilot stage. The challenge organizations face with AIOps are both technical and cultural: It requires a significant amount of time and effort to train and tune an AI model of each customer environment.

    [su_box title=”BMC Software Milestones”]

    • Dec. 2016: Peter Leav succeeded Bob Beauchamp as president and chief executive officer. Beauchamp remains chairman of the board of directors.
    • Oct. 2017: BMC Launches First AIOps Initiative for TrueSight
    • Nov. 2017: BMC Refinances Debt
    • Oct. 2018: BMC Software Goes Private After Being Acquired by Investors
    • Feb. 2019: BMC Partners with IBM on AI
    • April 2019: Bob Beauchamp resumes role as CEO
    • June 2019: BMC Extends AIOps Capabilities of TrueSight

    [/su_box]

    In addition, the effectiveness of any AIOps model varies based on the complexity of the IT environment. Organizations also should be concerned about the potential impact any flawed AI model might have on IT operations—it’s one thing for a human to make a mistake, but an AI model making a mistake at a level of scale could be catastrophic.

    From a cultural perspective, not every organization is yet comfortable with those AI models on current staffing, which can make some members of the IT staff less than comfortable with the whole concept.

    BMC CEO Bob Beauchamp

    It is early days for AIOps in DevOps processes. But the increased complexity of modern IT environments based on cloud-native technologies will force the issue within most organizations. BMC’s challenge is that while its management platform is employed widely in IT environments relying on ITIL-based frameworks, its footprint is comparatively nascent within organizations that have embraced best DevOps practices.

    Bill Talbot, vice president of solutions marketing for BMC, said one of the company’s advantages is access to the massive amounts of data required to train an AI model effectively. Most startup companies in this space don’t have nearly the same depth in terms of customer base, he said. That puts BMC in a better position to employ big data analytics to identify patterns and trends across multiple customers without having to know what data came from any one specific customer, unless the customer elects to share its data with BMC.

    BMC makes it a point to caution customers about rivals that make AIOps claims about their platforms but don’t actually solve a specific problem—a process known as “AIOps washing.” An AIOps platform not only needs to leverage Big Data analytics and machine learning, but it also provides the code and functionality needed to customize specific IT operations use cases, including capabilities such as automated event remediation and service resolution, optimized cost and capacity management and proactive vulnerability mitigation.

    BMC Software today stands at a crossroads. The company recently brought back former longtime CEO Bob Beauchamp to run the company day to day and has made no secret of the fact it is looking for additional investors. Making a pivot to AIOps requires access to massive amounts of funding for incumbents and startups alike.

    The next BMC act undoubtedly will involve employing AIOps across both DevOps and ITIL-based environments running on-premises and in the cloud. By any measure, that effort represents one of the most ambitious transformations ever attempted by any IT vendor.

    — Mike Vizard

  • Consider the Role of Programmers as Storytellers

    Consider the Role of Programmers as Storytellers

    Every time I speak with the programmers of our team in casual conversation, I ask what they’re working on. Every time, I get the same kind of answer, “I’m building a dashboard,” or “I’m optimizing a database,” or “I’m testing the user interaction.”

    In other words, I get the same answer as the proverbial masons, “I’m laying bricks,” or “I’m building a wall.” But not, “I’m building a cathedral.”

    At an abstract level, programmers understand the value of the big picture. But at a practical level, they focus on the immediate next steps. Since time is always crunched, programmers skip absorbing the big picture.

    “This is a mistake,” I told them. “Your work improves if you know why you’re doing what you’re doing.”

    “How?” the curious ones asked. That’s when I told them the story of Andre.

    Andre told me he was creating a dashboard. I decided to ask him the five whys.

    1. “Why are you building this dashboard?” I asked. “Because my team leader told me to,” said Andre.
    2. “Why does he need it?” I then asked. “Because the client Rudolph told him to build it,” he answered.
    3. “Why does the client need it?” I continued to ask. “Because his boss told him … I guess?” he replied.
    4. “Why does his boss need it?” I asked him. “I don’t know,” he said.

    I knew I couldn’t get further. So, I asked to see the dashboard. It showed 14 metrics and their performance across time. I had no idea what I was supposed to infer from the numbers.

    I spoke to Rudolph, the client, and asked why he needed the dashboard. His answer was similar, “Because Victor, my boss, wants to know how our operations are doing.”

    “So, do you need all 14 numbers? As a table?” I asked him. “Yes,” he said. “I need to know how each number is performing.”

    I tried a different tack. “Suppose you’re on an elevator with Victor, and he says, ‘Hi Rudolph, how’s it going?’ How will you answer? Are you going to recite all 14 numbers to him?”

    Rudolph thought for a while and said, “No. I’ll probably just tell him about what’s improved and what’s worsened. Then I’ll tell him what I’m going to focus on improving. That’s all that matters to him.”

    That left Andre and me wiser. Our aim was not to show 14 metrics. It was to help Rudolph and Victor figure out what’s improved and worsened.

    This helped Andre change three decisions.

    Change the Development Priority

    Not all metrics were equally important. Some were consistently stable. Some fluctuated a lot. He wrote the queries for the fluctuating metrics first. So, when the deadline hit and he wasn’t fully ready, he still had the most important metrics in place.

    Change the Design

    He added simple traffic-light colors to the metrics that grew the most and fell the most. It made it much easier for Rudolph to figure out what’s improved and what’s worsened. Rudolph never commented on it. But we knew without it, his response would have been, “There’s something wrong with this, but I don’t know what.”

    Change What to Optimize

    The queries run slowly. Rather than optimize all queries, Andre reordered the queries to render the fluctuating metrics first and optimized those. Those results appeared first, markedly improving how fast the dashboard appeared to load.

    Conclusion

    There’s a reason modern agile processes refer to user stories to capture requirements. But we lose the spirit of the story when we write, “As a user, I want to see the performance of 14 metrics.”

    The next time you write a requirement, don’t hesitate to write the full user story for the programmers. “When my boss is in the elevator and asks for a quick summary, I want to tell him what’s improved and what’s worsened, in less than 10 seconds.”

    Stories are memorable. They are easier to understand and communicate. They help explain why we’re doing what we’re doing. They also help prioritize development, improve the design and optimize better.

    Always ask for requirements as stories. Always narrate requirements as stories.

    — S. Anand

  • DevOps Is Killing the ITOps Engineer (and That’s Bad)

    DevOps Is Killing the ITOps Engineer (and That’s Bad)

    In theory, DevOps is about bridging the gap between developers and ITOps teams. In practice, however, DevOps often can feel as though its real goal is to turn ITOps engineers into developers, without really requiring developers to dive head-first into ITOps work.

    In this article, we’ll explain why DevOps gives more glory to development work than to ITOps, and how that poses a problem to the DevOps world as a whole.

    In DevOps, Developers Come First

    We live in a world that fetishizes code and the creation of code. Developers fetch among the highest salaries in IT. The work they do is sometimes compared to magic. There are a litany of boot camps designed to teach people to code. From Silicon Valley to The Social Network, coders are depicted in popular media as whiz kids who are building a fantastical world the rest of us can’t even imagine until the coders hand it to us.

    By comparison, ITOps engineers—people whose main job is to deploy and manage software, not write it—get much less glory. I’ve yet to hear of an ITOps boot camp. Generally speaking, ITOps employees get paid much less than developers; some ITOps workers, such as those who do help desk work, barely make minimal wage. And no one makes movies where an ITOps engineer saves the world by rebooting a crashed server or applying the latest security patches.

    This trend is problematic, of course, because the work performed by ITOps is equally critical as development work for sustaining our digital world. Without ITOps teams, all of the code developers get paid so much to write would be useless because there would be no one to put it into production and keep it running when it’s there.

    There are many possible explanations for our tendency to celebrate programmers much more than the people who manage the software programmers build. The most important reason, I think, is we live in a world where creators of all types are generally held in greater esteem than maintainers. That’s true in a variety of fields, from building science (where architects who design structures are held in much higher regard than, say, the roofers and painters who maintain them) to the automotive industry (the white-collared engineers who designed your car are probably paid a lot more than the grit-streaked mechanic who changes your oil). The IT industry is not special in this regard.

    DevOps as Euphemism for ‘Everyone Should Program’

    The fetishization of programming would probably happen even if DevOps didn’t exist. But sometimes, I think DevOps is exacerbating the tendency to deny ITOps teams their fair share of prestige.

    Beneath the surface of many conversations about DevOps culture and continuous delivery is the idea that if only ITOps engineers knew more about how to write code and discuss code with developers, their organizations would be better positioned to deliver software more quickly while also improving quality.

    In my experience, it’s much rarer to find DevOps advocates interested in pushing developers to play a more hands-on role in ITOps work. Almost no one goes around saying, “If only developers spent more time provisioning infrastructure, software delivery would be more efficient.” And when they do, they are accused of trying to kill the developer by requiring programmers to participate in maintenance-related tasks below their pay grade.

    I’d therefore suggest one of the challenges DevOps faces today is not killing developers, but killing ITOps engineers and further devaluing the work they do. To keep advancing DevOps, we need to pursue greater parity between developers and ITOps teams within DevOps culture.

    — Chris Tozzi

  • HashiCorp Tools Bridge DevOps, Legacy IT Environments

    HashiCorp Tools Bridge DevOps, Legacy IT Environments

    At its HashiConf EU 2019 conference this week, HashiCorp extended the reach of its Terraform Enterprise tools for automating the management of IT infrastructure beyond the realm of DevOps into legacy IT environments.

    Via integration with the Service Catalog from ServiceNow, IT administrators working with a traditional ITIL-based framework now can invoke Terraform Enterprise to provision IT services. HashiCorp CTO Armon Dadgar said rather than requiring organizations to hire an infrastructure engineer to automate the provisioning of IT infrastructure, IT administrators now can do the job.

    At the same time this week, HashiCorp announced that policy sets may now be configured using the source policies available in multiple version control systems (VCSs), including GitHub, GitLab and Bitbucket. The goal is to make it simpler to enforce policies that would limit, for example, the number of virtual machines a developer might be allowed to spin up within the context of a set of GitOps processes, said Dadgar.

    Finally, HashiCorp also announced that HashiCorp Consul service mesh networking software has been enhanced to include Layer 7 application traffic management capabilities alongside integration with the HashiCorp Nomad application and cluster scheduler software.

    In general, Dadgar said HashiCorp is moving toward enabling its portfolio of offerings to span multiple types and classes of IT environments. HashiCorp Consul, for example, can be run across both legacy IT environments as well as Kubernetes. Most organizations will need to support both legacy monolithic and microservices applications using a combination of DevOps and legacy ITIL processes, noted Dagar. Rather than having to master new tools for each environment, HashiCorp is making a case for a common set of tooling that can be applied across multiple IT classes of processes.

    While some organizations have fully transitioned to DevOps, many more organizations are still struggling to make the transition. Reasons for not making that transition faster range from simple inertia to a lack of skills. Rather than having to take an all-or-nothing approach to modernizing IT processes, Dadgar said HashiCorp is enabling each IT department to embrace IT automation wherever they happen to be on that journey, including being able to embrace new platforms as they become available.

    Dadgar noted that level of extensibility is critical because most tools are still being brought into organizations from the bottom up by individual members of the IT staff, rather than being championed from the top down. In the absence of an executive mandate, Dagar said the path of least resistance is to make IT automation tools as accessible as possible to members of the IT team regardless of skill level.

    There will, of course, come a day when IT leaders will try to rationalize processes around a common standard set of practices. The degree to which any IT organization has been able to achieve that goal, however, has been mixed at best. In the meantime, IT organizations will always need to keep their options as open and flexible as possible.

    — Mike Vizard