devopsy.kinsta.cloud

Tag: ITOps

DevOps and AIOps: Better Together

DevOps is an attempt to scale technology with humans but AIOps is the ultimate answer.

In today’s world, where so many essential business tasks have become digitized, IT teams must deal with constant change while ensuring zero downtime.

The irony is that, although IT has become business-critical, the productivity and agility of the people building and supporting the customer experiences has plummeted. This has serious consequences for businesses. When your apps and services malfunction, you lose customers and revenue drops. When IT teams focus mostly on fixing emergencies, agility suffers and innovation shrinks. You need seamless operations in order to grow, compete and thrive.

Historically, IT teams grew along with technology, beginning with the mainframe through distributed computing. But, the advent of virtual computing led to a world of microservices and ephemeral logic. Now companies simply generate too much data for humans to monitor and understand manually and with legacy tools.

This opened the floodgates for an AI-led operations evolution, known as Artificial Intelligence for IT Operations (AIOps). By streamlining and automating IT monitoring, AIOps has helped IT Ops teams regain control over IT environments, quickly detect and repair problems, and prevent outages.

DevOps: Back to the Future?

However, there’s recently been a shift in technology operations to once again attempt to scale technology with humans. The DevOps movement has placed the ownership, support and success of their services on the developers that write the code. This has decentralized technology operations teams that primarily operated with little context and unclear escalation paths, creating several smaller teams of DevOps engineers.

DevOps teams work together, each on their own microservice, toward the company’s desired customer experience and business goals. Depending on the operations model, there can also be a dedicated site reliability engineering (SRE) team or individuals on the DevOps team with the primary SRE role of monitoring observability data to discover potential issues. In this scenario, SREs feed the insights they uncover back into the development cycle to correct and strengthen the reliability and scalability of the DevOps teams’ services.

But, as incidents occur in real-time, the challenge remains with SREs and DevOps teams to achieve insights and awareness across their applications, infrastructure and, ultimately, business services. To understand where incidents are occurring and what impact they’re having on the services and customers, it’s necessary to surface important events from the noise, understand the relationships between alerts and obtain the context needed to engage the right teams and people.

This leads to another challenge: Getting the right people to respond and resolve incidents before there’s any business impact. Each DevOps team has its own responsibilities and tools. Often, DevOps teams don’t communicate except through their APIs. When incidents occur that need immediate attention from multiple people from multiple geographically-dispersed teams, how do you contact them and get them to collaborate, considering they have complex and different on-call schedules and escalation processes?

As companies continue to transform with a customer experience focus and a digital-first mentality, they realize the seriousness of these challenges, the magnitude of their monitoring data and its negative effect on their business. Part of transforming their business is transforming their technology operations: Enter AIOps with the light to shine on DevOps processes, leading to a manageable, efficient and profitable future.

What AIOps Brings to DevOps Teams

AIOps provides a unique solution to address operational challenges and cover every aspect of your service assurance strategy and business.

Bottom line: You need to free people to focus on mission-critical tasks and empower them to build better services for better customer experiences—not scale them along with technology in an attempt to operate it.

AIOps allows you to scale by integrating with the tools and infrastructure you’ve invested in over the years and by adding critical layers of AIOps (intelligence) throughout. By applying patented AI and machine learning algorithms to your observability and monitoring data, AIOps learns your environment’s normal behaviors and generates alerts accordingly. This must be done at the edge, close to where digital services produce vast amounts of data.

Based on the anomalies generated both locally and environmentally, AIOps will then surface and correlate important alerts from all sources into actionable, contextual insights. A true, comprehensive AIOps solution also identifies root causes and impacts, and prescribes potential solutions based on previous resolution steps and feedback. This all occurs in a virtual workspace, where team members can collaborate, visualize and provide feedback.

AIOps notifies and empowers the right people to take the right action. It simplifies and understands complex team structures, engagement methods, on-call schedules and escalation paths, ensuring the right people are engaged and collaborate, even when they’re located around the globe.

When a situation or incident is surfaced, it is sent to users in real-time. Based on the insights derived from the underlying data and machine learning applied, AIOps knows the exact teams and people that need to take action, and the ones that need to be informed. These users have the context they need to respond, share a consistent view and stay in sync throughout the lifecycle of the incident. Once the incident is resolved, a streamlined post-mortem speeds up future response, by using similar incidents and predictive analysis to spot repetitive and future problems.

AIOps: A Key Ingredient for Effective DevOps

The only way to scale with the technology being created in today’s and tomorrow’s world by DevOps teams is with AI. It frees humans to focus on creating and improving the customer experience that drives maximum profitability for your business.

May 7, 2020
DevOps for the Rest of Ops
The IT pro gig usually includes a basic headwind. They’re part of a cost center, and budget for resources and staff are limited by design unless they’re managing a strategic initiative, which—at least for a while—is more generously funded.

Early adopters of DevOps, Agile or site reliability engineering (SRE) teams have been budget beneficiaries when tied to transformation projects—as has always been the case for high-visibility IT initiatives. Fortunately, DevOps has proved more helpful and stickier than some other “next big things,” and for some organizations it’s the engine behind growth and even actual digital transformation success.

IT pros—pragmatic as always—have taken note of DevOps potential, even while keeping lights on, systems up and reliably delivering the critical services businesses have depended on for decades. The question for IT leaders is, how can IT teams realize some of the benefits of DevOps, while operating inside traditional IT organizations?

As DevOps has finally slid off the hype peak and into the shed of useful tools, experience has distilled out a couple of conclusions. First, the tenets of DevOps—breaking down walls, shared feedback loops and learning to love automation—don’t only apply to new or cool apps. They’re even more helpful in traditional IT. For example, the rise of automation isn’t driven by a conspiracy to replace admins with robots, it’s a necessity to shrink the toil of manually wrangling all the new complexity of cloud and cloud-native tech on-premises.

Second, many of the ideas behind DevOps and resulting practices are a direct attempt to fix issues in IT that have frustrated IT teams for years. Ask any IT pro if they’d like to move all deployment to Monday through Thursday, 9 a.m. to 4 p.m., instead of midnight Saturday nights. That team will go nuts for CI/CD. Better, it turns out you don’t have to do all the DevOps things to get many of its advantages.

But even if your business still operates from the C-suite down, entraining quarterly budget spending, waterfall models and lots of walled silos, there’s an opportunity to adapt DevOps processes and culture to deliver progress on truly transformative initiatives to help grow the business. The trick is figuring out which DevOps practices are valuable and which ones are more useful to developers.

Luckily, focusing on just four can get your team most of the benefits without (much) disruption, and development teams are eager to share its growing body of new techniques and successes. We’re working in an interesting epoch of IT history, a potential democratization of DevOps for everyone, DevOps for the rest of ops.

Continuous Delivery

Until recently, some IT pros were skeptical, even suspicious that the development teams’ goal has been to automate them out of a career. Others felt developers were demanding everyone to learn how to program full-stack and invent a way to move fast and break things just as they can (more safely) do. A few admins have even had unpleasant encounters with some DevOps proponents when they showed interest. Nobody likes reflexive replies, such as “You use Oracle and DB2? Switch to MongoDB and come back later” to eager 101 questions, even if they’re just from a handful of community members. It’s especially unpleasant for developers and IT pros alike because that’s not how operations works. We share, we’re open, we’re helpful. We came to tech to help end users get more out of technology.

Instead, successful teams often start by identifying tools supporting fundamental operations goals, without the distraction of any particular technology or vendor, and seek buy-in from both developers and ops. Because if you’re in operations, you’re the team that must:
1. Find a way to get software into reliable production.
2. Maintain functional, useful governance.
3. Maintain apps for years more than anyone thinks they’ll possibly be online.
Operations knows this because it has always been the last mile of delivery and deployment, it cares about availability and SLAs, and defines process for breakfast.

One of the first tools associated with DevOps is CI/CD pipelines, which are literally implementations of automated policy about how code goes from origination to production. But, overlooked in many DevOps blogs is the last step of CD because it’s not for the faint of heart. That’s unfortunate—operations is good at figuring out that last mile.

When operations teams embrace CI/CD/Continuous Deployment, they’re rewarded for investment in automation in the best possible way: less night and weekend work. As a result, the pipelines they build aren’t ephemeral experiments but production-grade CI/CD because it’s part of production itself. And as change velocity increases and error rates decrease, many teams refuse to go back to manual deployment unless they’ve exhausted all other options. Pipelines don’t replace IT pros, they improve work-life balance.

When the operations teams present developers with standardized, battle-tested nozzles through which changes get more quickly deployed with fewer errors, the reaction is usually excitement. It allows developers to mate their build, unit test, packaging and delivery head ends to the typically complex delivery/deployment operations pipeline. Best of all, ops teams tend to include monitoring, security and even cost governance into their processes, resulting in effective controls being automatically and reliably applied. Further, ops metrics create and enrich the feedback loops dev need to improve code over time based on real-world performance, not dev or test data. In environments like that, culture change is more organic based on shared goals, success and the flexibility to let each team do what they enjoy most.

Start Breaking Stuff Fast

Operations teams can have a little envy when it comes to breaking things. They’re technologists just like developers, and like most people attracted to technology, they like to learn by disassembling, recombining and otherwise hacking tech to learn the intricate details of how it works. But unlike a developer testing code, experimental configurations or other normal laptop environment one-offs, operations absolutely can’t experiment. IT is risk-averse by commandment, for good reasons.

“Get a lab!” sounds like an obvious suggestion, and it is—but it’s a great example of easier said than done. Because IT is a cost center, labs are often seen as duplication and waste because their value is harder to justify in a budget presentation. But with a little unconventional use of the monitoring data you already have, that’s not necessarily true. Operations regularly correlates change events with performance histograms in the ongoing search for improved reliability. Consider adding before and after metrics to changes tested in a lab environment first. You’ll likely find you can prove the more realistic the pre-deployment testing, the lower the error rate and higher the performance of that system.

But the mechanism behind improved quality isn’t the lab itself or data it produces. It’s the team’s experimentation—and some blood, sweat and tears shed—impossible in a production environment. IT also allows operations to invite developers to go on deployment and tuning ride-alongs. Developers can use whatever tools they prefer, make changes, probe and otherwise disassemble the lab stack as needed. And from a communications and culture perspective, this can be helpful.

Operations lab environments also allow IT pros to separate hype from reality on their own. They can install the latest shiny new tools for evaluation, try new methodology or even test significant platform architecture redesigns. In short, they can break whatever they want without affecting production and provide both feedback to developers and a place for dev to experiment with changes. Here again operations can be the lead, with new avenues for communication, cooperation and culture change.

Monitor Your Success

Most developers aren’t data driven, they’re functionally driven. They love data-informing design and are good with tools that wrangle it into something useful, but at the end of the day their job is to build code that does things. IT pros have monitoring in the DNA because detailed performance data collected over time is how they initially validate hunches but also break down conclusion bias. But while everyone seems to understand the value of metrics and data, it’s not usually considered a tool for DevOps communication.

As mentioned earlier, feedback loops are a fundamental principle of DevOps. Changes go into production, production generates data sent back to developers to guide future changes to further improve service quality. It’s a virtuous cycle, and both teams value it highly. The trick is in creating a language between dev and ops about what to collect, how it’s used and the stewardship of feedback over time. Step one is to sweep away preconceptions on both sides and start fresh.

A common approach is to bring a small—one-pizza small—team together and start with a blank whiteboard. The goal is to matrix the inventory of what’s already available with what’s helpful, then identify gaps for both teams. Development teams are hungry for the incredible richness of coverage operations can offer, while operations is eager to mine new instrumentation the development team relies on for troubleshooting and internal application performance monitoring (APM).

APM for hybrid applications is a great example of both teams learning how to create ensemble visibility into hybrid on-prem/cloud-native services. High performing operations teams ensure they add metrics dev teams need, and developers discover that the “impossible” information to debug pesky apps has been collected for years and is an API call away. Once again, a little conversation can lead to relief, and earned changes to hearts and minds.

Reward Success With Big Dreams

The final common experience of developers and IT pros both are never really done. Code can always be improved, and service delivery can always get better. But because we’re human, sometimes we get tired, stuck or bored, and it’s easy to take a break, especially after a big win. That’s natural. There are years of deferred work and technical debt lying around, and no time for perfectionism, especially in enterprise IT. This critical aspect of DevOps needs to remain a focus of any team adopting DevOps techniques—remembering DevOps is never done. It’s a culture of continuous, collaborative improvement and evolution. This shared understanding sustains the bonds between dev and ops.

When teams innovate, when they continue forging closer relationships and when they share improvements as the team culture changes, they start asking bigger questions. We’re conditioned in IT to accept that certain things can’t be achieved because, reasons. As a result, sometimes suggestions and questions that could finally resolve the team’s most bedeviling challenges might not always be made or asked.

Track your “impossible” ideas which drive progress, then promote successful ones while sharing details of why they worked. This in turn helps socialize the value of success resulting from openness and encourages great ideas that may have been sitting on the bench for a long time.

Again, you’re investing in more metrics and signals, but not limited to performance data. When you dream big, especially as you untangle the enormous expectations of digital transformation into operations changes, you’ll start to rely on new metrics. For example, business metrics tied to corporate objectives, behavior signals accurately reflecting CSAT, change and error rate data helping teams decide when to add new versus when to fix and new ways to assure leadership operations have everything under control. When managers are confident IT pros can safely transform gear, services and staff without interrupting the business, the most sought-after DevOps goal can come into focus: A new freedom to innovate at speed.

— Patrick Hubbard
April 2, 2020
How Enterprise Architects Can Help IT Operations
Traditionally, the role of IT operations has been to keep apps and systems running well for the business. But in the era of cloud, DevOps and disruptive technology, keeping the lights on isn’t enough. IT operations people need a broader view of business needs. They need to evolve the infrastructure stack in ways that will maintain stability yet also support business goals. This is no easy shift for people who’ve grown their careers by focusing squarely on the technology: procuring and configuring servers, storage devices and network components.

Enter the enterprise architect. Enterprise architects live and breathe business requirements and are a liaison between business leaders and IT. They’re responsible for building a long-term plan for technology evolution aligned with digital business initiatives. Enterprise architects are stewards of enterprise data and thus have typically established many solid relationships across not only IT but business functions as well. Typically, they are advisers to C-level executives and traditionally report to either the CIO or the CTO.

The chief architect can also be a valuable partner to the director of IT operations and his staff. Here’s how:
1. EAs bridge the gap with business units, helping define business services and their requirements which ITOps need to meet, and gathering end-user feedback which drives planning and staffing.
2. EAs bring a deep understanding of enterprise data assets, data connectivity and business applications—perspectives that help ITOps define key business service management goals.
3. Architects also assess the need for new infrastructure technologies and take a strong hand in evaluating vendor platforms.
The relationship between IT operations and enterprise architects may not be an easy one, since IT operations have been accustomed to working independently. They may not welcome other people/teams advising them on which monitoring systems to use, for instance, or how to prioritize incidents. Yet, better collaboration will ultimately help IT operations teams by improving their sometimes-contentious role with the business and providing a deeper understanding of the relationship between network and business outcomes. The following use cases illustrate how this can work.

Driving the Roadmap Between Current and Future State

One of the activities that enterprise architects conduct is creating a technology reference model: a list of all the technologies used in the business and notations as to which ones are approaching EOL and should be replaced. The EA’s job is to reduce organizational risk from outdated and misaligned tools and systems. On the other hand, IT operations managers have the viewpoint of managing risk; when a key infrastructure technology is suddenly jettisoned, that jeopardizes the health of business applications. EAs and ITOps teams have an opportunity to get on the same page, developing a transition plan together for moving to new technologies in a way that meets business needs and minimizes IT performance risk.

Improving the Stack for Business Value

As part of creating a long-term technology strategy, EAs focus on ROI. When ITOps share performance data on applications and systems, the EAs have critical data points to determine whether an application is too problematic–causing too many issues for too little business value–to maintain. Examples include moving from custom-built CRM to a COTS CRM or moving from on-premises email to Office 365 to increase availability and reliability of email.

Justifying and Evaluating New Systems

IT operations regularly want to purchase point monitoring software. They like what they see in a couple of new startups focusing on cloud monitoring and optimization or something specific for monitoring databases. But they may not know that another group, say in the EU region, is using something that could be applied to solve the issue. EAs can supply an holistic picture of the problem and how to solve it. In some cases, it’s not about buying a new system but leveraging an existing one to reduce complexity and minimize spend. EAs can also bring a business/user lens to assess user experience, such as AIOps. What effects could an AI application have on improving service delivery, and showing customers that IT is there to make work easier?

The EA and ITOps teams need to work together to support the business. Enterprise architects should bring to the table their understanding of business needs, existing technologies and their knowledge of technology investments that are being proposed outside of IT. Meanwhile, ITOps can bring granular information about device failures, common issues and preferred vendors, which helps guide the EA’s recommendations to business stakeholders. Together, these two critical roles can develop enterprise IT strategies which help the business succeed.

— Michael Del Castillo
March 16, 2020
A Deep Dive Into AI for IT Operations
Traditional IT administration and management techniques are reforming. By employing artificial intelligence (AI), IT operations are taking an interesting turn in the field of advancements. The term “AIOps” stands for Artificial Intelligence for the IT Operations. AIOps uses AI techniques and algorithms to monitor the data as well as reduce the blackout times. The dominance of digital businesses is introducing technological changes in their operations to streamline them and eliminate the overhead of traditional operational management.

Digital companies are generating abundant and information-rich data whose analysis and monitoring do not demand delays. From data collection to storage and data processing to analysis, all operations need to be frictionless to deal with them in an efficient manner. In the last couple of years, the AIOps market has exploded which is ultimately achieving the goal of reducing delays in problem-solving and identification.

Paving the Road to AIOps

The story starts with the need of AIOps and digital transformations. Cloud adoption and the shift toward other digital technologies are revamping the IT industry. It also includes the adoption of innovative solutions in the IT operations to make them frictionless. The technologies that are widely used in IT operations vary from AI to machine learning, big data to Internet of Things (IoT) and Application Programming Interfaces (APIs) for various use-cases to edge computing. All these innovations are penetrating into IT operations making them more smooth and sleek.

AIOps platforms enhance and automate IT operations, using the following different approaches:
- Using machine learning algorithms and neural networks for better analysis of gathered data. The data is gathered using IoT devices and tools connected with the source and sending data to the cloud for storage.
- The system automatically interprets if something is wrong in the operations. The errors are detected and solved immediately without introducing delays in the operations that are dependent on each other to complete.
- Reducing human power by automating the processes that include the effort of individuals.
Components of AIOps

The two basic components of AIOps that accelerate innovation in IT operations are machine learning and big data. Instead of spending human effort and time into cleaning, processing and analysis of the raw data collected from different platforms, IT operations are able to do all the tasks by employing machine learning algorithms and techniques to judge the nature of data and deduce necessary information from it.

Factors Boosting the Emergence of AIOps

AI is a promising technology that holds the confidence of building applications that can help the IT community. AI provides options for reliability, scalability and generic tools that can be deployed to promote the IT operations and boost them without utilizing human power.

Factors boosting digital transformation in the IT industry:
- Complex IT Infrastructure: Its operations face many challenges regarding data management and infrastructure. Manual IT operations are becoming harder to handle such as data collection, processing, cloud management, database efficiency, third-party management tools and SaaS integrations. Complexity management is a task that needs proper attention and the environment to deal in a better way.
- Handling Abundant Data: IoT technology is penetrating rapidly at an industrial level. IoT devices produce an abundant amount of data. Handling this much data is not the job of individuals. Therefore, by using AI-based algorithms the data is interpreted and analyzed. The performance is monitored using efficient algorithms.
- Increased Computing Power: The networking infrastructure has become robust. Therefore the IT industry is in dire need to enhance the processing speed also. By deploying IT solutions that harness AI and machine learning technologies, the computational power of the system can be increased.
Enterprises Using AIOps

The IT companies that are complex and large to which human power is not enough for overall operations are using AIOps to automate the tasks.

The adoption of technology has a huge impact on the success of IT companies:
- Development Teams: Companies that have a DevOps model and struggle to manage the roles between the organization are using AI and other innovative technologies to smoothen their processes. With this, developers increase the responsiveness, agility and clarity in the IT operations.
- Cloud Computing: Cloud computing has a big future. The companies store their huge data in the cloud and all the operations related to cloud data management need to streamline, as it has its own challenges. Therefore, to make the processes flow without errors and monitor the access, there needs to be automation.
- Digital Transformation: Companies around the world are all active in adopting technology solutions. Whether it is a fintech, travel, healthcare or e-commerce company, all businesses are automating their processing by first digitizing them and providing the user’s facility of utilizing applications based on newly emerged technologies. Information handling can be done by making operations handled by robotics and blockchain technology. AIOps removes friction from the IT operations.
— James Efron
March 3, 2020
Edge Computing and ITOps: Opportunities and Challenges

Edge computing is growing quickly, but IT operations need to evolve as well to effectively monitor and manage new devices, sensors, applications and data. Growing enterprise edge ecosystems should integrate and become part of centrally managed ITOM systems.

Edge computing is hard to define and is running high on the hype scale. But, research and surveys continue to indicate that this trend of processing data where it’s collected for better latency, cost savings and real-time analysis is an innovation with legs.

There will be 75 billion IoT devices by 2025, according to Statista. According to Spiceworks’ 2019 State of IT report, 32% of large enterprises with more than 5,000 employees are using edge computing, and an additional 33% plan to adopt it by 2020. Tied to the growth of edge computing is the advent of 5G wireless: 51 operators globally will start 5G services by 2020, according to Deloitte Global research from 2019.

The major cloud companies are also investing in the edge. The AWS Local Zones service allows single-digit latency connecting to computing resources in a metro environment, while Microsoft offers the Azure Stack Edge appliance and Google Cloud IoT is a “complete set of tools to connect, process, store and analyze data both at the edge and in the cloud.”

It’s safe to say that edge computing is becoming mainstream, and CIOs and their ITOps leaders should plan appropriately for it in 2020 and beyond.

Benefits of the Edge for ITOps

We’ve read plenty about the business benefits from edge computing: oil rig operators need to see critical sensor data immediately to prevent a disaster, marketers want to push instant coupons to shoppers while in the store, video security monitoring can catch a thief in the act and medical device alerts that ensure patient safety. Edge computing may save IT money on cloud and network bandwidth costs as data volumes keep exploding and the need to store every data point becomes harder to justify.

There are also implications for IT management and operations. Local processing of high volume data could provide faster insights to manage local devices and maintain high-quality business services when seconds make a difference—such as in the event of a critical server performance issue threatening the e-commerce site.

Today, ITOps teams are inundated with data from thousands of on-premise and cloud infrastructure components and an increasingly distributed device footprint. The truth is, only an estimated 1% of monitoring data is useful, meaning that it provides indications of behavior anomaly or predictions about forthcoming change events. With edge monitoring, we can potentially program edge-based systems to process and send only that small sliver of actionable data to the central IT operations management system (ITOM), rather than transmitting terabytes of irrelevant data daily to the cloud or an on-premise server where it consumes storage and compute power.

The job of filtering out the highly-contextual data on the edge, where business occurs, can support real-time decisions for successfully running ITOps at speed and scale—regardless of what combination of on-premise, public cloud or private cloud infrastructure is in place. At the same time, ITOps will need to be a leader when it comes to minimizing the risk of edge technology from a performance, security and privacy perspective. However, as detailed below, we are in the early stages of determining how to make this work in practice.

ITOps Realities for Edge Computing

Edge-Specific Security Needs Are Still Unknown: Edge devices are often small and infrequently designed with security in mind. More than 70% of edge devices don’t mandate authentication for third-party APIs, and more than 60% don’t encrypt data natively. So the attack surface in IoT and edge environments is now larger, and less secure. This is particularly worrisome when considering edge devices that collect personally identifiable information such as email, phone numbers, health data or financial formation such as credit card data. ITOps will need to work closely with security and legal teams to map out the company-specific risk, governance and compliance requirements around managing edge data.

Edge Monitoring Tools Are Immature: Companies need platforms that can instantly monitor and analyze edge-generated data. In the connectivity of tomorrow, billions of connected devices will be communicating machine-to-machine, and the addition or subtraction of connected devices will be possible at an unprecedented scale. In this environment, the ability to manage large volumes of connected devices and the information being exchanged between them will be critical. 5G acts as the unifying technology, bringing flow of information and the density of scale. We will see an influx of innovation in edge monitoring in the coming years.

New Environments Call for New Rules: As organizations move more data and application assets to edge computing environments, IT will need to devise new policies and thresholds for central processing and alerting of all this data. Applying AI-based automation is essential here, as manual efforts will have zero chance of keeping up with the volume of data filtering, analysis and response. We are entering the age of nano satellites, vis-à-vis SpaceX and OneWeb. These edge devices will transform the future of agriculture, energy, mining, transportation and finance due to their capabilities for sending insightful data in real-time to customers, wherever they are at any moment. ITOps will have its work cut out to understand and properly manage this evolving edge infrastructure.

DevOps Processes Will Become Even More Paramount: If you haven’t already realized that DevOps is taking over software development and IT management, just wait for when edge goes mainstream. There will be no other way to manage change and deployments of edge technology without the agile, continuous integration and continuous delivery methodology of DevOps. It will be imperative for ITOps to adopt DevOps practices and tools to manage, monitor and deploy edge resources.

ITOps is at a crossroads, determining how much of the past is still relative and how much they will need to change to adapt to a distributed, hybrid cloud world that will soon include edge as a fundamental pillar of their digital strategy. Security, machine intelligence and DevOps will be crucial areas of expertise for ITOps teams looking to help drive better business value and customer experiences from the edge.

— Bhanu Singh

January 22, 2020
Predictions 2020: Key Trends that Will Shape the Autonomous Digital Enterprise

As 2019 comes to an end, and we look toward trends driving innovation in 2020 and beyond, technology advancements in hyper automation, AIOps, edge computing and the convergence of IT service management and IT operations management will be at the forefront.

Autonomous Digital Enterprise

In order to thrive in the 2025 futurescape, organizations will need to evolve to Autonomous Digital Enterprises (ADE), a state where intelligent, integrated, value-creating functions can operate with minimal (ideally zero) human involvement or interference, across every facet of the organization, and its ecosystem of partners.

This concept is based on the notion that every company today is being disrupted by technology and that’s fundamentally changing their business model. The notion of every company being a technology company means that software becomes more than just a system of record to be the system of action, the system of insights and the system of engagement.

Edge Computing and IoT

Over the next couple of years, partnerships will prove to be the pathway to success in realizing value from edge computing and IoT solutions. By using seamlessly integrated, multi-vendor, commercial off-the-shelf solutions, enterprises will experience more success than they would with single-vendor niche products. At the end of the day, what we need is an ecosystem of integrated solutions as opposed to a single silver bullet. This ecosystem will be able to harness data and look at it in meaningful ways and translate information into insight that then translates into actions that businesses can take to optimize the use of IoT assets.

Convergence of IT Service Management and IT Operations Management

As IT service management and IT operations continue to converge in 2020, CIOs will rely on AI. More than ever, CIOs will need to address this convergence and work to break down silos between these departments. To obtain a comprehensive view of the IT landscape, CIOs will begin to adopt AI, cognitive and robotic process automation. This will bring end-user support and managed back-end infrastructure more closely together.

If we look beyond traditional IT and service management, in the near future we will have the ability to go to market with fully integrated solutions that enable boundary-less operations across AppDev, SecOps, ITOps and service management. This will further enable the use of DevOps in organizations using the power of AI to improve existing capabilities and create new capabilities. This doesn’t mean companies have to start from the ground up on AI efforts, rather they need to look for software solutions that can offer prebuilt AI/ML solutions that drive change.

AIOps

The demand for AIOps in the enterprise will continue to rise as AI and machine learning have taken the industry by the jugular. Due to an expansion in the number of workloads—both in public cloud and on-premises—and an increase in application complexity, investment in AIOps will increase and ultimately lead to better business outcomes. Today’s challenges place a premium on differentiated vendor solutions powered by AI/ML and big data analytics techniques that can help modern IT operations evolve from traditional monitoring to observability to actionability.

All of these trends open realms of possibilities for what businesses can achieve in 2020—as long as they can harness the massive amounts of data enterprises have access to and the technology that will allow them to unlock those insights.

Want to learn more about what to expect in 2020? Join us Jan. 23 for our Predict 2020 Virtual Summit featuring discussions from some of the industry’s best and brightest offering up their visions for the future. Sign up today for this free daylong virtual event.

— Ram Chakravarti

December 24, 2019
The Role of Containers in Business Transformation
Containers hold new promise for business and digital transformation. Yet many differences exist between monitoring containers and monitoring other elements of infrastructure, such as VMs, storage, memory and compute. Understanding those differences is vital for ITOps managers as they navigate their businesses’ digital transformation.

Interestingly, the differences are as much technical as they are organizational. Companies need to explore these differences and consider questions such as, how to ensure that a container-based project performs when it goes into production, and what changes you must make to infrastructure to ensure you have enough compute and storage power to support the containerized applications your development team is going to launch.

The essence of container management is to be prepared for the changes coming from application development, which means understanding resource and infrastructure as it is today to estimate the effect of those changes tomorrow.

Here are four differences ITOps managers need to consider with container management as they move their digital transformation journeys forward.

Containers Versus VMs

The signature feature of containers is they can hold and run an entire application all in isolation. VMs offer operating system level isolation, but at the cost of a much larger footprint; to begin with, each one has its own copy of the entire operating system. VMs are also far less portable than containers, especially when moving among private and public clouds and data centers. As a step toward digital business transformation, most companies are looking for DevOps tools and for ways to deploy their applications to the cloud, whether for QA or production. Organizations rolling out new applications are also gravitating to a container-based architecture from the start.

Advantages of adopting containers include:
- Agility: Instantiating a new container goes as much as 20 times faster than instantiating a new VM. Startup is measured in seconds rather than minutes and once deployed, containers scale up and down rapidly in response to load.
- Density: Most containers are megabytes in size rather than gigabytes. With their smaller footprint, containers allow for as much as five times greater density than VMs.
- Cloud-readiness: Containers can be deployed to any cloud (private, public, virtual or physical) which opens up the possibility of using the same containerized application in hybrid and multi-cloud environments.
- Blast radius: As software developers incorporate and release changes more quickly, they look for ways to control potential damage from bugs. A problem in a container is less likely to cause widespread damage and is easier to isolate and repair than a problem in an entire VM or physical server.
- Cost savings: Every VM requires a hypervisor and an operating system (OS), which can involve licensing and fees. Containers run side by side, sharing the OS kernel of a single physical machine while holding isolated copies of specific library versions they depend on. Architecturally, VMs tend to contain more components and require greater allocation of resources. Containers, on the other hand, tend to hold a specific component or a single micro-service, minimizing resource overhead.
Containers, DevOps and Real-time Monitoring

A company adopting DevOps to speed up deployment of its software products automates as many development processes as possible and relies heavily on automation scripts to move code along, from continuous integration through continuous deployment and continuous delivery (CI/CD/CD) to production. Along the way, multiple developers must check in on new changes. Automated processes, such as health checks and unit testing, take place and then the build is deployed to the next environment, such as integration or QA. With each set of changes, ITOps needs the answers to several questions:
- How will this affect performance?
- What will be the impact of the changes when we deploy?
- Do we have enough compute, storage, memory and networking to support the changes?
In a traditional environment, developers have plenty of time to answer those questions. But if they’re deploying modern, containerized, cloud-based applications, that window tends to be much smaller which can lead to the kind of guesswork that makes ITOps uneasy. That’s where container monitoring comes in. When ITOps can see real-time and historical analytics of containers and their hosts, across physical, virtual and cloud environments, they can create performance benchmarks to help them make informed choices about infrastructure.

Microservices Get Added into the Mix

Microservices are loosely coupled and independent services, each with their own release schedule and lifecycle. Developers are able to make small changes to elements of these applications which allows them to mitigate the risk of negative business impact by reducing the blast radius around those changes. In case of failure in a single change, there’s less to repair and less impact on surrounding, dependent services. We’re seeing developers move away from packaging a single application per container to packaging a single service of an application per container. This allows them to make changes more frequently and keep up with business demands.

Multiple Silos

The goal of DevOps is to bring application development and traditional ITOps closer together. However, container monitoring and management changes things. Traditionally, areas such as storage, network bandwidth, VMs and physical hosts that require monitoring have all been in the ITOps silo, an area that developers don’t typically own. Developers write code and build software in the application development silo and then hand the software off to the ITOps silo to run. However, things have changed and now developers aren’t only writing code, but also owning and using containers in the application development silo. Once the software and containers leave this silo, their consumption of resources like compute, storage, network bandwidth and memory will also affect the ITOps silo. That’s why monitoring and management is as necessary for containers as it is in areas like storage, VMs and physical hosts.

While container technology may be new for the traditional enterprise, the impact it will have on business transformation cannot be ignored. Containerization brings increased simplicity, consistency and portability to production environments, enabling faster IT delivery and increased performance, arming businesses with a new agility and allowing them to rapidly respond to changing customer needs. In fact, according to 451 Research, application containers will be a $2.7bn market by 2020.

The businesses that embrace this opportunity will be taking a vital step towards transforming into bustling digital enterprises that can deliver innovative products and top-notch customer experiences. To learn more about containerized infrastructure and cloud native technologies, consider coming to KubeCon + CloudNativeCon NA, November 18-21 in San Diego.

— Yinghua Qin
October 28, 2019
From Pets to X-as-Code: Enabling Control and Predictability for ITOps at Scale

Amy Wheelus, vice president of network cloud at AT&T, recently spoke about declarative predictability at the Open Networking Summit in Antwerp. AT&T is the world’s largest telecommunications company providing mission critical communication services where people’s jobs–and lives–can be on the line. As they move away from labor-intensive IT operations toward cloud-native automation, there is a need to trust the declared IT infrastructure state can be predictably controlled and reproduced. This requirement to build predictability, reproducibility and trust into IT systems is not new but has grown in both scale and complexity as IT departments went from a few servers blinking in the basement to the core around which every business rotates.

With every customer interaction and business function becoming more digital, companies must be able to rely upon their IT departments if they hope to continue to keep pace in today’s rapidly changing world. This article traces the evolution of IT operations from Pets towards X-as-code with each improvement bringing companies closer to declarative predictability. These best practices enable enterprises to reliably deliver core business value through their IT department and keep pace in the cloud native era.

Yesterday: Pets Versus Cattle and Infrastructure as Code

Back when the IT department was just servers in the back room, it was all hands on deck when anything went wrong because work ground to a halt. The CEO couldn’t get their emails and the sales department couldn’t book new orders. From these early days, it was learned that in order to trust IT with critical business services, high availability and reliability needed to be built into the system. This lead to the creation of the Pets vs. Cattle analogy where each infrastructure component is interchangeable and can quickly be replaced when anything goes wrong. This allows companies to quickly recover from or even avoid outages.

While treating servers as cattle solves some problems, as any rancher can tell you, handling a whole heard brings its own set of challenges. As the size of managed infrastructure grows, it can be difficult to keep track of where, when and how it is actually deployed. In addition, changes to production systems to keep them healthy can lead to configuration drift creating unique snowflake servers which are pets rather than cattle. It becomes a difficult task to predict how a system will react or rely upon it in critical situations when its actual state is unknown.

To help eliminate this problem and control how infrastructure is configured and deployed, ITOps teams moved to infrastructure-as-code. In this way of working, applications and servers are never modified once deployed. If something needs to be changed, a new version is created and the old one is decommissioned. Each deployment can be versioned to create an overview of what is in production and how it is deployed.

Managing cattle rather than pets and using infrastructure-as-code has helped companies move from ticket-based operations with long lived machines to more scalable and dynamic infrastructure. These practices build reusability, scalability, automation and versioning into IT systems to create a simpler, more predictable deployment and management process. By being able to declare the state of its infrastructure, companies are able to predict and trust the outcome.

Today: Cloud-Native Infrastructure

Looking at the definition of cloud native from the Cloud Native Computing Foundation, it quickly becomes clear why many companies are turning toward these best practices. They help companies build and run resilient and manageable infrastructure that–combined with robust automation–creates dynamic, yet predictable systems. Declarative APIs and immutable infrastructure will be used as two examples to understand how cloud native builds upon and expands the techniques covered above.

Declarative APIs make infrastructure as code and automation possible. For example, with Kubernetes, operations teams are able to declare the desired state of the system and automate the reconciliation with the actual state until they match. It originally began with just container orchestration but has since expanded to cover many other types of cloud resources including the physical infrastructure itself through Custom Resources and Cluster API respectively. Declarative APIs make infrastructure management simpler and more reproducible by automating operations and eliminating errors in production. Declarative APIs can be leveraged to create immutable infrastructure.

Immutable infrastructure goes beyond just servers as cattle and only treating the infrastructure as code. Immutable infrastructure covers every higher layer of the stack until each one is codified and managed like cattle. Only exact copies are deployed and they can be completely replaced at any time. If anything needs to be changed, a new deployment is made and the old ones are decommissioned. Immutable infrastructure works for the whole stack to eliminate variations leading to fewer deployment failures, consistency across environments, easy horizontal scaling and a simple rollback and recovery process.

As these two examples show, cloud native enables companies to reliably scale out dynamic IT services without having to simultaneously scale out their operations team. Building declarative predictability into systems engenders trust in automatic processes, even when critical services are running on them.

Tomorrow: X-as-Code

While cloud-native best practices provide many benefits, applications and infrastructure are just the beginning of the story for enterprise IT teams. In today’s world, they are presented with the Herculean tasks to manage a dynamic and sprawling environment while still keeping in compliance with internal policy along with national and international laws. This challenge moves far beyond mere servers and treating them like cattle to compliance and governance across multiple clouds and hundreds or thousands of developers. With these challenges in mind, the question still remains on how to best deliver on these demands at the enterprise scale.

Looking across architecture, configuration, blueprinting, policy and governance, if each of these is simply viewed as another layer of the stack above the application, their operations can be radically simplified. By applying the principles of immutable infrastructure and cattle to this further layer of the stack, each of these can be defined and managed using an X-as-code model. These types of management models can already be seen taking root across the cloud-native ecosystem.

Kubernetes creates blueprints-as-code through the use of Admission Controllers while policy-as-code can be defined in Pod Security Policies and Network Policies. The CNCF even has a policy engine, OPA, as a sandbox project that gives operators fine-grained yet flexible policy control across their entire stack. By using code to decouple policy and other layers from the actual implementation, the resulting stack is easier to understand, flexible enough to handle future requirements and less expensive to maintain.

At Loodse, our customers were struggling to manage configuration, services and policy across their clusters. Leveraging the Kubernetes controller paradigm in Kubermatic, we are able to automate the deployment and lifecycle of each of these add-ons. This gives users the ability to declare, enforce and automate the required state of not only the infrastructure but the configuration and policy, too. Kubernetes controllers bring predictability and control to dynamic and scalable cloud-native IT even as it begins to handle more and more mission critical services.

Cloud Native in Production: Declarative Predictability

With IT becoming the core of every business, bringing cloud-native best practices into production enables operators to predictably declare, run and manage their infrastructure at the enterprise scale. Being able to trust in these automated operations frees up resources and opens up many new possibilities. Developers can be given guardrails and become empowered to do their job rather than waiting in a ticket queue. Operators can easily create highly available self-healing infrastructure that meets governance standards allowing them to focus on delivering a platform with differentiated business value.

At each step of the development of modern IT practices, from cattle to X-as-code, operational processes have become more streamlined allowing both developers and operators to drive more value to the bottom line. The future of IT infrastructure and operations will be a trusted platform that automates undifferentiated heavy lifting, allowing the whole business to focus on customer value.

To learn more about containerized infrastructure and cloud native technologies, consider coming to KubeCon + CloudNativeCon NA, November 18-21 in San Diego.

— Bill Mulligan

October 25, 2019
Chaos Engineering for ITOps

Chaos engineering (CE) is the discipline of experimenting on a system in order to build confidence in the system’s capability to withstand turbulent conditions in production. This approach is becoming commonplace in software development and operations (DevOps) practices. But how would its application extend to ITOps? CE for ITOps offers a similar framework for stress-testing a technology platform to understand its weak points and performance pitfalls under heavy pressure.

CE tends to be used primarily in DevOps during bug testing: setting up experiments to run software under different conditions, such as peak traffic, and monitoring how it functions and performs. This becomes increasingly necessary in cloud-based systems where failure to understand extreme load responses could result in runaway cascade failures or, worse yet, spinning up thousands of extra nodes handling error conditions while not doing any actual work. These same principles, applied to IT operations management (ITOM), help define a functional baseline plus tolerances for infrastructure, policies and processes by clarifying both steady-state and chaotic outputs when extremes are reached.

Applications in IT

The theory of CE in DevOps gained early traction at Netflix as they moved from physical to virtual infrastructure, with the team that implemented it on AWS breaking off to form Gremlin. However, CE is not typically used in ITOps, because ITOM has historically been separated from development (generally, IT monitors system dynamics, and when a problem occurs, engineering change management or ITSM is brought in to remediate the issue).

With the growth of containerization in cloud applications today, IT infrastructure looks more like development environments than classical multi-tier architectures. But the limitless scale of the cloud means failures can also be limitless: Microservices are well-served by testing elasticity and scalability, data flows and resiliency through stressing the system to the edge of its tolerances and fixing their shortcomings before a public crash.

One, Two, Three…Chaos

Implementing chaos engineering for ITOM provides a systematic approach to identifying weaknesses in a microservices-world. In a monolithic environment, you have visibility into performance and event metrics that may be lost with microservices designs. As a result, the need for operational insights becomes even more critical when scaling to unknown workloads.

Netflix’s Chaos Monkey grew out of CE principles from their own cloud-native community, meant to address the gaps in common dev tools’ abilities to manage extreme complexities. This methodology is extendable to infrastructure and helps to set guardrails on platform behavior as a whole.

Here are five fundamental steps to follow in order to bring this thinking into your team’s ITOM.

Define the Current Steady State

Performing baseline analysis is a standard concept in capacity planning, upgrade strategies and other high-impact functions. Start with something relatively simple (and small) so you don’t get overwhelmed by the data, or risk interfering with the business if something goes wrong (such as security Red Teaming). For example, monitoring CPU and network utilization, which are common bottlenecks in any IT shop.

Define Optimal Conditions

There’s how your system generally operates, and then there’s how it should operate; these typically aren’t the same thing. CPU utilization and network latency are always affected by application efficiencies, hardware conditions and a host of other factors. Create a standard that outlines what engineers should expect on a normal day, on an easy day and on a very hard day. These are the control groups, and the extreme day will be the stress test.

Form a Hypothesis

Where will the system break? If you’re running an application scenario such as doubling the peak traffic that even your worst day so far has seen, will your CPU maintain optimum utilization (or will the container provisioning engine smoothly deploy additional nodes) as in the variable control groups, or will it spike so severely that processes grind to a halt because there isn’t enough memory or network bandwidth left to manage the load?

Execute a Real-World Event (But Contain the Blast Radius)

Do something extreme, such as taking down a firewall that severs connectivity to one internet service provider. This will confuse the application as it tries to respond to requests with repeated failures, ramping up CPU processes as errors return from a dead network endpoint. Log events will mount, filling the database and saturating the backbone.

Validate the Hypothesis

What happened? Monitor utilization and network throughput during the test and see where the system fell over. Is it what you expected, or did something never previously considered take place? Did new chaos erupt from the fissures in your infrastructure? Stabilize, document and remediate.

Never Stop Not Being Afraid

Stressing a system to its absolute max—and a little bit further—to see where things go wrong allows you to understand steady-state behavior and error-handling, so you can fix it before something breaks in new and unexpected ways. What do traffic spikes look like? What are real-world events and their impacts on your organization?

CE is not just for DevOps. It should be a systemic practice for load-testing (out of your comfort zone) to the point of failure. It’s a responsibility for more than microservices deployments and applies to all sorts of disciplines within the IT organization.

— Michael Fisher

September 30, 2019
Why ITOps Is Moving to the Cloud
There’s been a lot of movement in the IT operations management (ITOM) business lately, from the acquisition of SignalFx by Splunk to the PagerDuty IPO, and all signs point to a Datadog IPO in the future. What’s with all this consolidation? I believe we’re seeing the rise of a future-state of ITOM; that is to say, it’s the rise of SaaS-based ITOM. And it’s easy to see why.

In my previous consulting career as lead enterprise systems architect, our team had an impeccable record in designing and implementing well-architected hybrid infrastructure solutions. We maintained an immaculate record and near-flawless customer satisfaction record. By project sign-off, our job was always done. And yet, returning to the same solutions six months later told a different story entirely.

Well-architected solutions are similar to human bodies: They are perfect when they’re born but need constant care and feeding. These same solutions that satisfied SLAs, exceeded expectations and transformed organizational efficiency can easily degenerate, and just like our bodies have their nervous systems to monitor, brains to send alerts and tissue to self-heal, well-architected systems need operational maintenance to keep them humming.

The traditional approach for solving this eternal need was and still is to design and implement well-architected IT operation management (ITOM) solutions, around a well-architected infrastructure. And yet, this is a self-fulfilling paradox because the ITOM solution itself needs the same care and feeding.

There’s a Problem On-Premise

ITOM is a broad term encompassing application and operating system (OS) performance, alerts, log management, notification, asset configuration, incident management and more. It typically involves the purchase of a suite of on-premise point tools addressing each need, and then to develop an internal framework to help those tools interoperate in a meaningful way. While that is possible conceptually, the facts on the ground reflect a very different reality:
- Multi-vendor tools are often not designed to work together.
- Creating an internal logical framework that orchestrates various teams and technologies can be very complex in large enterprises.
- It’s near-impossible to create technical integrations flexible enough to adhere to inevitable organizational and technological changes that will affect this logical framework.
- Predicting cost-of-ownership is nearly impossible since each tool is controlled by a different vendor, and the internal integration effort is often unknown.
- Predicting the cost of the manpower required is also very difficult, as each tool requires its own set of specialists, in addition to integration specialists to make it all work together.
- Upkeep is often overwhelming, as vendors offload software patches, upgrades and on-premises hardware costs to the customer.
In the face of all these challenges, the end result is often unrealized value, overwhelmed operational teams, loss of service and inability to accommodate new technologies resulting business service disruptions.

Why Cloud? Why Now?

It’s historically been near-impossible to build a traditional ITOM platform on-premise. Vendors typically sell a collection of white-labeled tools cobbled together by acquisitions, and this is far from a platform. The complexity and the rate of technological change make it difficult to provide consistent quality and value across the various product lines. This puts the ITOps team squarely in the middle of a pickle: How can they ride the wave of a changing environment without relying on static tool suites?

The Future is Flexible

Enterprise ITOps has been stretched now more than ever. There is a serious skills gap, shortage in IT workforce and ever-increasing technical complexity. Time and resources are precious, and Enterprise ITOps need simplicity and predictability along with flexibility and control. Enter SaaS ITOM. By moving the ITOM function to a SaaS orientation, the responsibilities, workloads and daily tasks can transform according to the needs of the organization:

Keeping Up with the Business

SaaS ITOM can keep up with technological change, and keep pace with cloud, DevOps, artificial intelligence and more. In the world of SaaS, change is an accepted constant and not an inconvenience. What’s more, SaaS ITOM is infinitely more consumable than the tool suites of legacy past, and that reduces the learning curves associated with running ITOps.

Keeping Up with Industry Needs

A SaaS ITOM platform will be able to deliver a framework that’s both flexible and governed and can accommodate technical and organizational complexities. This agility is a feature of modern SaaS. SaaS ITOM can also integrate features running on a single code base supported completely by the SaaS vendor, who will absorb maintenance and upgrade cycles, freeing considerable and valuable time back to the operator. All of this results in a more predictable total cost of ownership, improved service quality and more value to the business user.

It’s not news the world of IT is moving to the cloud. It is news, however, that cloud can offer such transformational benefits in ways we’ve never seen before.

— Wael Altaqi
September 4, 2019