devopsy.kinsta.cloud

Tag: IT operations

  • What NetOps Teams Should Know Before Starting Automation Journeys

    What NetOps Teams Should Know Before Starting Automation Journeys

    Automation has captured the human imagination since the earliest days of recorded history. A fanciful example of this comes from Homer’s classic “The Iliad” in the form of a bronze robot called Talos, which was designed to automatically monitor and defend the island of Crete from marauding intruders. The appeal of automation for IT organizations, including network operations (NetOps) teams, may not be so life-and-death but it’s undeniable, nonetheless. More and more NetOps teams are turning to automation to achieve higher network uptime (by reducing manual errors), increase efficiency and reduce costs and help deliver a better and more consistent operational experience for all users.

    Due to the allure of these promised benefits, the temptation for NetOps teams to integrate automation as soon as possible may be strong. However, it is important for these teams to remind themselves that automation is a tool, not the destination itself. Indeed, simply overcoming the technical challenges of deploying automation may not result in the successful outcome teams want, without identifying and defining the intended outcome of what their automation journeys should be.

    This article will explore some practical steps and best practices that NetOps teams should use throughout their own automation journeys.

    Getting the Journey Started on the Right Step

    A well-known maxim in business is that you should visualize success and start with the end in mind when embarking on a major project or initiative. This is a good reminder for NetOps teams as they begin their automation journey. That is, these teams should be able to answer the question, “What is our desired end state when it comes to automation?” before they even evaluate what tools or technologies they may want to consider.

    Teams who have this mindset will have a clearer understanding of their overall mission as well as an enhanced view of the various projects, processes and products they can improve through automation. It is critical for NetOps teams to take the time to put in the work of defining the desired IT experience at the end of the automation journey, even if the task itself may be difficult. For example, this experience may help define how NetOps engineers will interact with an automation tool through its code, its dedicated user interface, or even its generic service portal. As the saying goes, “no pain, no gain.” The teams who invest in the time and effort to plan ahead will be far more likely to maximize the benefits of automation.

    What to Automate?

    In addition to visualizing what success looks like, it is important for NetOps teams to identify what tasks or processes they should be automating. This is important because there is a vast universe of NetOps tasks that can be automated. Taking the time to identify tasks to be automated will generate huge benefits for the NetOps team as it will help them select the correct automation strategy and toolset that will deliver them the results in an efficient way. And while specific projects may be unique to each NetOps team, in general, there are two broad categories of tasks that are ideal for automation:

    1. Highly repetitive tasks. These are manual or semi-automated tasks that are not necessarily hard but can be cumbersome for NetOps teams. Examples of repetitive tasks that can be automated include assigning an interface into a specific VLAN or perhaps updating network security policies or rules.

    2. Highly complex tasks. These can be more strategic and difficult tasks that can be prone to human error due to their scope, scale, and complexity. Automating these tasks may require additional planning and steps to ensure success. Examples of complex tasks that are ideal for automation include service redirection from one device to another or BGP peering with third parties.

    Think Big, Start Small With NetOps

    Like all people, NetOps professionals enjoy the results of a job well done. So, while the vision of their automation journey may be big, it’s important to start with a small, short-term project that can be completed quickly. There are a couple of benefits to this approach:

    1. Quick automation wins will give NetOps teams confidence for future projects.
    2. Projects like this can generate data and feedback that NetOps teams can convert into learnings and insights for the next project.

    This approach can also be applied to bigger, more complex automation projects. Instead of taking on the entire scale of the project at once, NetOps teams can break it down into smaller components. For example, infrastructure-as-code (IaC) is a broad category of automation entailing the managing and provisioning of systems through machine-readable definition files. There are many considerations to large-scale IaC projects such as identifying the use case, e.g., Day 0 or Day 2 operations. The advantages of this approach are the same as with the quick-win scenario: There is a better likeliness of success and more immediate feedback and data to guide the NetOps teams through this entire process.

    Finally, as talented as most NetOps teams are, they are not likely to have all of the automation expertise in-house at any given time. So, it is important that they address skills gaps (automation, programming) through strategic vendor partnerships, internal/external recruitment and additional training for their teams. Again, instead of recruiting or investing en masse, NetOps teams are well-advised to take the time to plan, execute, learn and modify as they go along.

    Summary

    The human fascination with automation is stronger than ever. The industry buzz around one particular form of automation—especially artificial intelligence and machine learning—is indicative of this. NetOps teams have much to gain from investing in the tools and skills required to do automation right.

    Automation of data center network pods, self-contained zones that represent a section of the overall data center, is a good example of a strategic project requiring forethought and planning to execute well. A logical first step in this process might be to anticipate how to manage future changes when automation is in place. Options for this may include code manipulation, Git (open source code management) or CI/CD pipelines, all elements of infrastructure-as-code automation. Or it may involve improving user experience through a specific UI interface or through a self-service portal. The next step then might be to decompose this large configuration into smaller bits starting with the registration of the new switches into the controller. Once this is done, we can now go into port configurations and then eventually move toward larger and more sophisticated elements such as the automation of VLAN assignments and L3 peering.

    For long-term and large-scale automation projects like this, it is especially important to follow the recommended key steps outlined in this article, which are:

    1. Start with the end in mind. It’s critical that the teams take the time and effort to define what success looks like, especially in the form of the IT experience they want to achieve.
    2. Know what you want to automate. There is an endless supply of tasks and processes that can or should be automated. Prioritizing automation projects that NetOps wants to take on is a major milestone in any automation journey.
    3. Think big, start small, learn as you go: Quick automation project wins are key to building confidence and “muscle memory” for future projects. For bigger projects, the best approach is to break them down into more manageable projects that can provide vital data and insights during the journey.

  • Building a Platform for DevOps Evolution, Part Two

    Building a Platform for DevOps Evolution, Part Two

    In the introduction to this series, we discussed the concept of change and how it applies to DevOps environments and release managers. 

    In a world of constant changes in our surroundings and user demands, DevOps has to maintain hundreds of moving pieces across large organizations of people who, by nature, are adverse to change. This requires release managers to delegate power, transition from a centralized commander to an enabler, understand different value streams and constantly experiment. 

    In this section, we’ll cover some of the ways and types of experiments DevOps organizations can conduct to improve their release management procedures and meet their objectives using DevOps principles.

    Value Stream Mapping

    Value stream mapping is a series of steps used by organizations to provide products or services their customers want or need. In the traditional sense, the release and deployment process can seem tedious, especially if teams must communicate with a release management team to schedule release slots on a calendar.

    Despite its advantages, value stream mapping has a few drawbacks. It requires a lot of people and it can be difficult to coordinate and sparsely repeated.  

    Value stream management platforms can extract the four key DevOps metrics for teams to inspect and adjust in real-time. However, there are some shortcomings with using the four metrics:

    • Lead time is characterized as time from code commit to live; however, in value stream management, we also consider the time it takes for an idea or customer request to reach value realization.
    • Deployment frequency becomes interesting when teams deploy sporadically but are still improving, but when teams attain the ability to release on demand, other metrics become more helpful.

    As a result, teams must alter metrics over time to match team ability. This is why it’s critical to empower value stream teams to discover and measure their own metrics. Another factor is that people are more likely to change when doing it for themselves rather than having it done to them.

    Lessen Batch Size

    Reduced batch size lowers risk and reduces the time an activity waits in a queue. When changes are minimal, they may be implemented quickly and with less danger of causing disruption, thereby making the distinction between the phases less significant.

    While some key metrics may be accessed through specific tools such as backlog or release automation, without a value stream management platform, it’s more difficult to gain insights into overall queueing time.

    The value stream map is automated when all aspects of the DevOps toolchain are combined, resulting in a unified system from idea through the realization that allows teams to understand their cycle time and value flow efficiency and observe how changes improve work.

    Evolving Architecture From Monolith to Microservices

    Most organizations have monolithic, tightly coupled applications because that was the best software design approach available in the past. However, as the industry evolves, release managers become increasingly conscious of the limitations of this architectural approach:

    • Large batches of work are required to build, test and deploy.
    • Dependencies must be managed, including people and systems.
    • Dependencies cause fragility.
    • Fragility causes bureaucracy (e.g., CABs).
    • Bureaucracy and handoffs hinder flow.

    Rearchitecting applications to microservices, like changing human behaviors in a system, does not happen overnight, and there are frequent bottlenecks along the way. This is often a “change or die” conversation. If the application does not become more capable of change, market disruptors will provide new experiences to consumers, and customers will jump ship for these competitors.

    A value stream management platform can reveal where restrictions, delays, and dependencies occur and how they might be reduced. As a tenet of DevOps says, “Don’t manage dependencies; break them.”

    For many companies, however, the road to systemic autonomy, or loosely coupled systems, is lengthy with many incremental steps of improvement. This method is recommended by the strangler pattern: chipping away at the monolith one bit of functionality at a time.

    During the application architecture change, we need to support value stream teams, so they don’t waste time on unexpected work caused by unknown dependencies. We also need them to see and appreciate the improvements they’ve made with the support of a value stream management platform.

    Part of the release manager’s job is to identify value stream success stories and share them with the other value streams so that local improvements can become global ones.

    Transfer the Release Management Role to the Value Stream Team

    When teams work independently on small changes, they can release them as soon as they are ready. However, moving from one state to another takes time to transition to that stage of the journey.

    In a traditional method of working, the release process is likely to be coordinated by a release manager or a team of release managers. The release manager role can be transferred to the value stream team, with the team providing access to systems that provide visibility into current releases to the release manager.

    Transparency into operations across value streams provides traditional release managers with great comfort during this transition, relieving them of the administrative strain. It allows individuals to learn new skills and refine the meaning and purpose they derive from their work.

    By taking an end-to-end and cross-value stream approach to change and improvement and ensuring blame becomes accountability, release managers can be key to the success of these improvements. Platforms for value stream management give the data needed to support the conversations that will lead to this cultural and behavioral shift.

    In this series’s third and final part, we will cover more types of experiments and DevOps best practices for achieving change and driving results. 

  • NewOps? AIOps? NoOps? Just Don’t Call Me Late for Dinner

    NewOps? AIOps? NoOps? Just Don’t Call Me Late for Dinner

    Once again, we are seeing an uptick in the use of the term NoOps. I always felt that NoOps was a misleading and, in fact, empty phrase. I don’t care what magic you think you have; you are never going to eliminate the need to operate your software, applications and infrastructure.

    I first heard the term NoOps early in the maturation curve of DevOps. Back then, the idea seemed to be that Dev was going to absorb Ops. Between that and the cloud, there really wasn’t a role for traditional Ops to deploy. Managing apps would be easy with the cloud and automation. At least, that was the pie-in-the-sky thinking—but I never bought that.  

    There were other terms that popped up, like NewOps and AIOps, though that became something else entirely. But really, I think the rise of the systems reliability engineer (SRE) role put the NoOps movement in its grave (where it probably belonged). SRE became the shiny new identity of the Ops crowd and SRE continues to be one of the most popular topics here on devopsy.kinsta.cloud.

    This latest incarnation of NoOps seems to be a little different. From what I have seen, it is not really NoOps; instead, it is more of a “not your Ops to worry about” movement. It is shifting who is responsible for Ops from the company developing and using the apps to a service provider (or SaaS provider) who is responsible for Ops. It is more a “MyOps” not “YourOps” movement, but there is still most certainly an Ops there.

    In my neighborhood growing up, we used to have a saying, “I don’t care what you call me, just don’t call me late for dinner.”  Originally, to me, NoOps was very much like that. We changed the name of something, but it didn’t go away or fundamentally change the thing itself.

    Now, we still have Ops, it is just a question of who is doing it. That is not NoOps to me. It is “somebody is doing MyOps for me.” 

    We saw a similar pattern in security. The idea of outsourcing security was never called “NoSec,” because the idea of no security is, frankly, downright scary. Instead, it just meant relying on a cloud vendor or an MSSP to handle your security for you; it was outsourcing your security. An important lesson learned from that was that you can outsource your security ops, but you can’t escape your security responsibility. That is to say, when your customers’ information is breached, saying “We had someone doing this for us!” did not absolve you of responsibility.

    The same thing holds true for this new “OutsourceOps.” The outsourcer may do a better job than you at Ops. They may give you SLAs and SLOs that give you reasonable expectations of performance and remedies if those SLAs and SLOs are not met. But it is still your company’s name and reputation on the line.

    So is what we seeing as NoOps really NoOps? No, it is Outsourced Ops. But as we said in the old neighborhood, I don’t care what you call it, just don’t call it late for dinner.

  • Why We Still Need Specialists

    Why We Still Need Specialists

    A few years back, I wrote here that specialists—particularly infrastructure specialists—were not going anywhere.

    I think it is worth updating that prediction and reminding you all that specialists across the bulk of DevOps subdomains are still necessary—even if you have broken down their silos.

    Spend time in any of the variety of places that DevOps folks hang out and listen to/read what they actually say. When they express their sentiments as a real-time activity, the idea that we made one team responsible for most, if not all, of an application’s life cycle seems rather brutal.

    In most industries, we have increasing specialization as complexity increases. Even if it was allowed, most carpenters are not qualified to do plumbing or electrical work. Oh, most know the basics and could do it in a pinch, but they aren’t the expert that the plumber or electrician is. IT, for the last many years, has tried to do the opposite, with a decent rate of success. But practicality wins out over rigidity to make that success.

    We still have specialists. And you should ignore people that suggest they need to go away. The situation is already bad enough without having a person you can turn to for help with a harsh issue in this language, or on that piece of networking gear. While some of this—language specialists are the easy example—is merged into a DevOps team, some of them—network admins, for instance—only partially are. The networking within/between containers is often handled by the DevOps team (using one set of specialties), while the networking that allows all of the organization to run is still handled by operations. But I can’t think of an org I deal with regularly that doesn’t house these specializations somewhere.

    This is one of the things that has traditionally held back low-code/no-code solutions—they assume these specializations and will require specialists in whatever the underlying technologies are when things inevitably go wrong. The same is true (to a lesser extent) of DevOps, AI and automation. Someone has to know what is going on underneath—they must have specialized knowledge—in order to fix it when things go wrong.

    Remember, you are asking your DevOps teams to be both Jack of all trades and master of some. That is a recipe for failure, but DevOps is designed to withstand or overcome failure in short order, so I only mention this so you understand why new hires seem wild-eyed at times and old hands seem jaded—there is a lot going on that they’re mastering and remastering.

    So, stand up if you’re one of the “sailing against the wind” crowd. Be proud you’ve managed to thrive in an environment with a ton of specialist parts and make sure you know who to ask for help when something goes wrong and you end up out of your depth. Even if the answer to “Who can I ask for help?” ends up being “Learn it for yourself and then be prepared to help others,” you’ll rock it either way and the rest of us who have struggled through it feel your pain and respect your work. So, thanks—for those who’ve never experienced it, but rely upon your work.

  • Best of 2021 – Seven Reasons You Shouldn’t Hire a DevOps Engineer

    Best of 2021 – Seven Reasons You Shouldn’t Hire a DevOps Engineer

    As we close out 2021, we at devopsy.kinsta.cloud wanted to highlight the most popular articles of the year. Following is the fifth in our series of the Best of 2021.

    DevOps is a culture, not a title. Therefore, companies should follow DevOps principles, rather than hiring one savant to fix an entire organization’s delivery model.

    I recently spoke with Uri Zaidenwerg, DevOps lead at Replix, to understand the current state of DevOps. As more robust DevOps tools emerge and practices proliferate across organizations, developers are now better equipped to own their applications end to end, including deployment, security and even cloud costs.

    Extracting DevOps from the development cycle could defeat the purpose of de-siloing development and operations. Below, we’ll outline some reasons why you should focus on building a holistic culture instead of filling a DevOps engineer role.

    1. DevOps Is a Culture, Not a Role

    The first reason is also the most encompassing — DevOps is not a title; it is a culture. DevOps functions best as a mindset adopted across IT. Instead of hiring a dedicated role, all “developers should be doing operations,” said Zaidenwerg.

    This means giving developers better visibility into DevOps KPIs, such as deployment frequency, deployment time, change failure rate, time to detection and availability, which we’ll explain in detail below.

    2. Developers Know Their Application Code Best

    Developers know their code better than anyone else. Diagnosing errors in an unfamiliar environment is tricky, and only programmers truly comprehend the ins and outs of their own code. If armed with DevOps skills, they could address problems much more efficiently than external eyes.

    Zaidenwerg worked as a DevOps systems administrator, and described “what it’s like to have to monitor availability of an application that you did not code or build.” It’s difficult; simultaneously diagnosing multiple applications that you don’t know intimately is also quite challenging, he explained.

    If you are unable to link errors to application processes quickly, it could hurt availability and SLAs. Instead, encouraging developers to diagnose issues could reduce time to remediation because they have more context, noted Zaidenwerg.

    3. Developers See the Surrounding Environment

    In addition to understanding how individual applications function, developers also have unique perspective into the surrounding environment — they realize the intricate network of dependencies and interconnections that make up a microservices system.

    To see what’s going on, you really need to know how to code. “DevOps engineers need to be software developers,” said Zaidenwerg. Handing off DevOps tasks to an administrator who doesn’t have coding knowledge can severely limit their ability. They will not be able to dig into the code to address real issues or make pull requests confidently.

    4. Separating Roles Keeps Ops in a Silo

    Relegating DevOps responsibilities to a centralized team goes against a fundamental principle: de-siloing departments. “DevOps was invented to make that gap smaller,” said Zaidenwerg. Having a separate role would thus appear to contradict one of the fundamental tenets of the methodology.

    “The point of a DevOps practice is that developer teams are empowered to control their own operations, not that you have specialists who know how to script operations,” said Tom Marsh, principal developer at Novartis, on Quora.

    5. One DevOps Engineer Can’t Fight a Flood

    As it stands, there are aren’t enough DevOps engineers to go around. According to answers on Quora, the ratio of DevOps engineers to developers tends to stand between 1:10 to 1:12. Zaidenwerg said he’s seen worse — from 5:50 to 6:120. Instead, he said, “Zero to zero is the optimal number.”

    One DevOps engineer against a tidal wave of requests and application deployment pipelines is just not sustainable. “Can you imagine the amount of features and code that has been written?” asked Zaidenwerg. “The amount of fires they need to put out is huge.” And what happens when that single DevOps engineer goes on vacation? To avoid delays, he encourages developers to own their CI/CD process from end to end.

    6. What Happens When DevOps Engineers Automate Their Job Away?

    Another creed of DevOps and SRE practices is to introduce automation – with the ultimate goal being to automate their job away. But, if this were actually to occur, what happens to the DevOps engineer? Do they sit back and rake in the dividends of some upfront work for years afterward? It would make more sense to have programmers introduce automation into their workflows.

    7. K8s + Other DevOps Tooling Empowers Developers

    Powerful DevOps tools like Kubernetes, ELK stack and cloud infrastructure solutions now make it far more realistic to delegate more responsibility to the programmer themself. Democratizing “as-a-service” solutions for monitoring, load balancing or storage enables everyone to “share the burden of the DevOps work,” said Zaidenwerg.

    Many organizations have begun to treat Kubernetes as a source of truth. Zaidenwerg wholeheartedly encourages this practice. “Everything should be managed on K8s, from K8s,” he said. “Try and think of K8s as your lowest layer.” Standardizing microservices around K8s helps retain platform-agnosticism for multi-cloud, he added.

    How to Get There? Track DevOps KPIs

    So, if we aren’t delegating DevOps tasks to a DevOps engineer, how can we encourage developers to embrace DevOps principles? To Zaidenwerg, the answer boils down to how we define success. Instead of gauging success using traditional development output metrics such as productivity, scrum or number of tasks completed, we need a culture that measures developer work using DevOps key performance indicators (KPIs).

    There are many DevOps KPIs to consider. Some could be:

    • Deployment frequency: How often do you commit?
    • Deployment time: How long does it take from committing to the live environment?
    • Automation: How automated is the build process?
    • Availability: Is the program globally available and highly performant? What is the latency and uptime?
    • Monitoring: Are you proactively monitoring and fixing issues in live environments?
    • Optimizations: Is the application cost-effective? Can it be more efficient? Are there idle resources running?
    • Policy as code: Are security policies automated as code? “Security needs to be written in code … if it’s done manually, you’re f**ked,” Zaidenberg said, bluntly.
    • DRP: Is the application resilient? What is the disaster recovery plan (DRP)?

    Of the above KPIs, optimization stands out, as rising cloud costs are an ever-present hurdle these days. Zaidenwerg encouraged developers to track cloud environment expenses. When you have a daily graph of spending, you can use this visibility to “create supporting systems to automatically take idle resources down,” he said. Others have described this as BizDevSecOps.

    Encouraging DevOps Culture

    Our cheeky headline is not meant to devalue those with DevOps in their title — on the contrary — hiring someone with this rare experience could greatly assist organizations in introducing DevOps principles. They could act as a model to help cultivate this culture.

    In addition to tracking KPIs, Zaidenwerg recommended a top-down approach to spread DevOps principles. The methodology must be sanctioned and led by the CTO, architect or tech lead, depending on the company’s size. A DevOps culture is especially paramount for SaaS or IaaS companies, whose architecture must be more than error-proof —  “It must succeed every single time,” Zaidenwerg stated.

    Disseminating modern cloud best practices is also critical for building a DevOps culture. When you transition from hardware to the cloud, you often encounter similar tech (load balancers, firewalls, availability zones, etc.), yet with different names and implementation processes for each function, complicated by nuances between cloud service providers (CSPs). All this will require “translating on-premises best practices to cloud best practices,” described Zaidenwerg.

    Another significant issue arises from the fact that DevOps isn’t really taught in college-level computer science courses. Looking to the future, Zaidenwerg (and myself) hope to see more emphasis within the education system. Instilling a DevOps mindset early on would better prepare developers for more real-world settings encompassing deployment, availability and security-as-code.

    Whose House Is It?

    Zaidenwerg likens a software application to someone’s house. The homeowner knows every detail — where the utensils are, how to work the shower nozzle, what day trash is picked up, etc. Invite a stranger into a new house, and they will be at a loss.

    Similarly, inviting external Ops to manage a foreign infrastructure will put them at a loss. “If you let someone else deal with infrastructure, then you have two parties and no one is fully aware,” Zaidenwerg said.

    To summarize — DevOps is a culture, not a role. Treating DevOps as a role, instead of as a practice, could defeat the purpose and cause siloed, slower-moving teams. Instead, tracking KPIs and empowering developers with DevOps tools could help them keep their house in order.

    Of course, this is an evolving practice, and each organization will have its own understanding of what it means to implement DevOps. Now, do you still think it warrants the title? Please let us know in the comments below!

  • Log4j Puts Effective IT Operations at Center Stage

    Log4j Puts Effective IT Operations at Center Stage

    News of the Apache Log4j vulnerability exploit is striking fear into the hearts of both software makers and users. Log4j is the most popular Java logging service used today, with over 400,000 GitHub downloads and has been embedded in most internet services and products from companies all over the world, including Apple, Amazon, Cloudflare, Steam, Tesla, Twitter and Baidu, just to name a few.

    Exploiting this vulnerability is simple and allows threat actors to control Java-based web servers and launch remote code execution (RCE) attacks. New variations of the original exploit are being introduced rapidly—over 60 in less than 24 hours. The attack surface grows by the minute. Good credential management and advanced authentication will not help as this exploit is pre-authentication, which means an attacker doesn’t need to authenticate to your web applications to attack. Your web server is vulnerable.

    The good news is your organization likely has a team of seasoned IT operations pros that can help protect against the worst of Log4j’s devastation.

    How IT Operations Can Protect Against Log4j Vulnerabilities

    Step One: Find Log4j Wherever it Exists

    You need to investigate every internet-facing application, website and system that you own or use. This includes self-hosted installations of vendor products and cloud-based services. Focus on systems that are internet-facing that contain sensitive data. Once you’ve completed assessing your hosted apps and vendor systems, move on to endpoint applications. Java-based apps like WebEx, Minecraft, Citrix and many more.

    Step Two: Patch, Patch, Patch

    Patch in the same order as above: Internet-facing applications, websites and systems. If no patch is yet available for your apps and systems, uninstall.

    The pandemic has forced many endpoints into workers’ home environments, which creates extra challenges for patching those endpoints and keeping them compliant. Operations teams can’t rely on work-from-home folks to patch their own systems, even if they’re given clear instructions. Many of them will just ignore those directions, so you need to adopt systems that enforce continuous compliance.

    Step Three: IT Operations is Essential

    Communicate with your senior leaders. They should be in the know about this one! Senior leadership needs to position their IT operations teams among their most vital employees. Don’t think this is going to go away any time soon—we’re just starting to get a glimpse of what is being tried out there in the wild. Buckle up. 

     

  • Where Do SREs Go From Here?

    Where Do SREs Go From Here?

    Charlene O’Hanlon talks with Leo Vasiliou, director of product marketing at Catchpoint, about the results of a study the company fielded with VMWare Tanzu and DevOps Institute of nearly 300 site reliability engineers (SREs). This year’s report underscores the challenges of multi-cloud, calls out the underutilization of AIOps and shows a systemic shift in core baselining data. The video is below, followed by a transcript of the conversation.

    Announcer: This is Digital Anarchist.

    Charlene O’Hanlon: Hey, everybody. Welcome back to TechStrong TV. I’m Charlene O’Hanlon, and I’m here now with Leo Vasiliou who is the Director of Product Marketing over at Catchpoint. Leo, thanks so much for joining me today. Really appreciate it.

    Leo Vasiliou: Oh, of course, Charlene. Thank you very much for allowing us to have the conversation today.

    O’Hanlon: Absolutely, absolutely. I want to talk to you about a recent survey study that you guys put out in conjunction with VMware Tanzu and DevOps Institute. What were you guys looking at?

    Vasiliou: Well, from an existential perspective, we hope that the survey that we do leads into the annual report and we’d like for it to be the most data-backed set of insights in its community. There are sort of three underlying constructional essences of the report. One is to provide a core baselining data for the key SRE tenants. So DevOps, who had an amount of time, would call in reported levels of toil. The other essence is to look at what’s trending. And then the third one will generally be around the IT to business relationship. And then with each one of those constructional essences, the context and line items within each section will change from year to year.

    O’Hanlon: Gotcha. Alright. So this one focuses on site reliability engineering and what they’re seeing and what they’re dealing with on kind of a daily basis. Is that right?

    Vasiliou: Yes, yes.

    O’Hanlon: Okay. Alright, great. So tell me a little bit about some of the findings in the study. I know you guys do these, as you just said, reports yearly. So I’m interested in knowing what’s changed from year to year and what really surprised you in this year’s report.

    Vasiliou: Well, I like to say SREs are double struck with challenges because the SRE is essentially two jobs in one, right? They have to increase the efficiency of their operational activities while mitigating the risk of their transformational activities. So one of the key findings this year, which by the way this may be the most provocative dataset we’ve published just because some of the findings conflict with some popular theories in the market, but take the first one just as an example of that, is as part of the baselining set of data there was a self-reported drop in toil, which is the word they use to define repeated, mundane, busywork that doesn’t have any inherent value, of 15 percent. And one of the reasons this is such an insightful nugget to lead us into our various conspiracy theories is it was across the entire distribution of data, across all of their percentiles, as well as across all of the geographies that we looked at.

    O’Hanlon: Wow.

    Vasiliou: And so for me, when I had the blessed opportunity to do the research and sit down and co-write the report, having been stuck in the house for, let’s just call it a year and a half at this point, there was just something different about working on a piece like this and it just felt a bit more meaningful. And the conspiracy theory there or the hypothesis is that if people return to work, or return to the office I should say or some type of hybrid environment, will those reported levels of toil increase again next year as people adjust to the hustle and bustle of commutes just adding to frustration in how they think about things?

    O’Hanlon: That is a really, really fascinating thought process there. We could do an entire conversation about just that and whether what happens this year is going to be indicative of what happens in the future or if this year is going to be like a reset year. So from an SRE perspective though, the fact that the level of toil decreased 15 percent, that seems pretty significant. If you look at the entire spectrum of what they have to deal with on a daily basis, 15 percent, good lord, that’s huge in the SRE space.

    Vasiliou: Yeah. It is huge. I agree. It’s also based on the response of hundreds of survey participants year over year. And what’s interesting, Charlene, is one of the unique tactics we took in this year’s report is before we publish the reports and post survey pre report, we actually showed the results to various industry, let’s just call them, leaders. One of them we shared this particular section to had an alternative theory on why levels of toil dropped, and it had to do essentially with the fact that if people are not in the office, maybe does the very act of them not being observed compel them to do less busywork.

    So there was an alternative theory. But bringing us back to and leading into one of the other essences of the report, which is kind of the trends piece, it had to do with the amount of monitoring data sources, like the volume, variety, velocity of data, and the number of multi same-service platforms, e.g. multi CDN, multi DNS, et cetera, in use. And so will the return to work and the promise of AI ops help them keep toil levels low? Which leads us into the next sort of surprising finding, which is that SREs and their implementation or shift to AI ops is quite slow to maybe what we might be reading about in the industry, which was probably the second most provocative finding in the report.

    O’Hanlon: Yeah. So why do you think it is so slow then? Were there any trends that kind of came out of that? Any reasons why organizations are kind of slow to adopt AI ops?

    Vasiliou: Well, there is obviously a deluge of information and rhetoric in the market on the internet today. We need to seriously consider this idea of an information diet, which is a concept I first heard about a couple years ago. When we think about the idea of the hype cycle versus the challenges SREs face on a day-to-day basis, sometimes the market and the promise of what these types of solutions will offer might outpace actual implementation capabilities from, hey, we’re an SRE, let’s go implement this. I think part of the challenge here is probably the concept of AI ops is too broad and too large for consumption. It’s the idea of just saying, “Hey, let’s go solve world hunger or, hey, let’s go solve or implement AI ops,” right?

    O’Hanlon: Right, right.

    Vasiliou: You’ve got to break it down into smaller components and incrementally develop capabilities from there. Otherwise, how the heck are you going to baseline your progress for when you’re done? I think it’s too big, too broad, and some of the vendor promises may be of self-healing, auto remediation, maybe they were a little more on the hype side, where some of the other AI ops of components like root cause analysis or even correlation, maybe those are truthfully more on the hope and promise side.

    O’Hanlon: Okay. Do you think that AI ops – to your point, it is broad – do you think it just means different things to different people and so somebody actually may be implementing some type of AI ops and actually not realizing it?

    Vasiliou: The short answer is yes. So when we asked the question what monitoring tools or practices are used today, they were presented with seven options, ranging from application, infrastructure, experience monitoring, AI ops, and it was the fifth lowest, meaning that almost 40 percent of respondents said they’d never used it in the first place. The second question we asked is very simply what is the received value from AI ops, where one was the low value, nine was the high, and the range evenly spanned the value scale that was presented in the survey. What was interesting is there were a subset of people who said they never used AI ops but that they still received value, which was an additional dimension we didn’t think about when writing the survey.

    But it kind of brings me back to the essence of your question, which is are you user versus a consumer? And then go back to the previous comments about how big and broad this quote/unquote AI ops idea is, and there’s absolutely a propensity that people hear and think completely different things, unfortunately.

    O’Hanlon: Interesting. Interesting. So one of the things that we’ve been talking about in the SRE space over the last couple years is really how the role is kind of morphing and changing along with the market in general and to address a lot of organizational needs as well. Are there any findings from this year’s study that kind of bear out how an SRE role is changing and even if it is changing?

    Vasiliou: That will go back to what we were saying just a moment ago about mitigating the risk of their transformational activities. When you were asking the question, I’ll just cite the stat from our very first SRE report. We said that 85 percent of respondents said that in order to be successful in this role you have to commute into an office. So that got topsy-turvy turned upside down on its head.

    O’Hanlon: Right. Yeah. [Laughs]

    Vasiliou: But to get back to the essence of the question here, when we looked at the quantity of data sources and the prolific rise of the use of multiplatform providers, we sort of looked at this correlation between how SRE orgs are structured. Are you decentralized, are you centralized, or is there some type of hybrid in between? And then the decentralized piece was broken down even further by stack component, by platform component, or by business product or service. The vast majority of SREs said they are currently decentralized by business product or service, but I think the transformational piece here, which is something we’ll look at next year when we write the report, is because of the prolific rise of multiple third party and multiple same-service third party providers, there is a huge, gigantic spotlight to shift the primary dimension of org structure from product and service to a platform basis.

    And we talk about it like this. If you’ve got 10 different teams assigned to 10 different products or services and one member from each of those SRE teams developing capabilities that happen to be the exact same capabilities in development by those other nine teams, that’s the scalability and efficiency _____ that we start to lead in and talk about in this year’s report. So I think the transformational piece will be the assignment of SRE development engineering resources from product and service as a primary dimension to a platform – for example, cloud, CDN, DNS, API – as the primary dimension where those capabilities will be normalized and developed. Even though underlying platform structures will be different, those capabilities will then be offered as part of, let’s call it, an internal marketplace for use by multiple teams to kind of get away from that concept of silos and being segmented by product or service. So a bit of a long answer but all contextual to the key part there, which is the trend and the transformational piece that we wrote about this year.

     O’Hanlon: Right. Yeah, well it really is food for thought there if you consider how the SRE role was established and how it’s kind of – if you go down that path, how it seems to be changing and evolving as the more technology is adopted and more things are moved out to the edge and the need for SREs at basically all points on the network if you think about it. So great stuff there. Well, Leo, I know we could talk about all of the findings and have a much larger conversation, but unfortunately that’s all the time that we have for this one. But I do want to thank you very much for going through some of the survey findings. If folks are interested in taking a look at the study itself, is it readily available?

    Vasiliou: It is, it is. So it’s available at Catchpoint.com. Just head over to our learning and research section and enjoy the read. Charlene, one of the sentiments we wanted to leave readers with this year was this concept of baselining against Google’s definition of SRE, but we hereby declare by the power vested in us by absolutely nobody at all that you are hereby free to be an SRE.

    O’Hanlon: Oh, wow. Just one more thing to add to my plate. [Laughs] Well Leo, thanks again. I really do appreciate your time, and it’s a pleasure talking to you. I’m sure that we’ll talk again in the future about future studies that you guys have going on. So, thanks again. Really appreciate it.

    Vasiliou: You are quite welcome. Thank you for your time, Charlene.

    O’Hanlon: Alright, everybody. Please stick around. We’ve got lost more TechStrong TV coming up, so stay tuned.

    [End of Audio]

  • How Low-CodeOps Can Transform DevOps

    How Low-CodeOps Can Transform DevOps

    Over the past year, enterprise adoption of low-code software development platforms surged across the world economy as COVID-19’s economic consequences drove an unprecedented number of digital initiatives. Enterprises are turning to low-code platforms to rapidly build new apps that empower customers, remote workers and external end users conducting business operations or pivoting to new opportunities.

    Accelerating software development, however, only solves half the puzzle of a digital-first approach. Quickly getting solutions into users’ hands requires careful planning to determine how these new applications will function within the organization’s overall software operations strategy. This is often conducted across multiple cloud deployments.

    But the real prize comes when low-code’s underlying principles of automation and abstraction are extended into the DevOps domain of software maintenance and management. The ability to push out a continuous pipeline of software updates with the click of mouse, with streamlined testing and quality assurance, requires a platform that automates myriad, time-consuming tasks associated with building, testing, releasing and maintaining applications in an enterprise-grade production environment. This inevitably leads to the concept of “low-code operations;” or, to coin a phrase, “Low-CodeOps.”

    Indeed, unless the benefits of abstraction and automation — in this case, primarily speed — extend to operations and deployment, the real benefits of low-code development are elusive. A platform that lets you build an app in six days but requires six months to deploy is no solution at all.

    Harnessing the Cloud for Game-Changing Results

    To witness the vital role seamless cloud operations played in business innovation, look no further than biotech giant Moderna and its success in Operation Warp Speed. Moderna used the public cloud to pull together geographically dispersed collaborators, integrate scientific insights and refine its production cycle, enabling the company to deliver its first batch of mRNA vaccine samples a mere 42 days after the COVID-19 virus genome was sequenced. The time-to-value for this solution was definitely warp speed.

    And, it’s safe to assume, expensive. Cloud engineers and DevOps system administrators are highly skilled specialists who roll up their sleeves at 2 a.m. to fix unexpected glitches buried deep in the connected ecosystem of software operations. They figure out how to make the software actually run. Enterprises, no doubt, wish they could hire more of them.

    But the same shortage of skilled IT professionals that fueled today’s demand for low-code software development also impacts DevOps. In this case, the same low-code platform is the solution, empowering a wider cross-functional, collaborative team to reduce operational burdens — in a word, Low-CodeOps.

    Essential Features of Low-CodeOps

    To achieve a frictionless handoff when moving applications and data to the cloud, a production environment leveraging low-code principles will deliver on these key principles of low-code operations:

    Performance monitoring: Daily monitoring and log checks of platform infrastructure are the backbone of modern DevOps operations, ensuring that complex interdependencies are running smoothly. Automated dashboard monitoring can transform hours spent on these repetitive tasks into opportunities for engineering teams to focus on innovation, while still ensuring the health and performance of enterprise software portfolios. The tooling needs to enable automated end-to-end visibility into application workloads and performance characteristics of upstream and downstream services.

    Compliance: Every economic sector is subject to specific government regulations. Software platforms and DevOp teams must keep track of every requirement when integrating new functionality into software upgrades. Banking customers must deploy digital solutions on infrastructure that’s earned ISO/IEC 27001 certification. Healthcare solutions must conform to federally mandated HIPAA requirements. Defense contractors must be FedRAMP certified. Data—the lynchpin of any digital solution—is subject to privacy laws that differ across nations and borders. Imagine having to manually recheck every iteration of your application deployments for compliance. Luckily, low-code operations mean having an automated, system-wide compliance framework in place. This makes short work of service organization (SOC) audits and reporting.

    Security: Cyberattacks that lead to system outages and disruptions are a growing concern in our digitally connected world. Well-resourced cloud giants are not exempt from damaging cyberattacks, such as last year’s SolarWinds hack. Resilience is key. Ongoing security scanning can halt automatic software deployments when suspicious changes are detected. Prevention is always best. Monitoring tools identify vulnerabilities, enabling systems to quickly come back online and help prevent outages that can last for hours or days.

    Infrastructure independence: Widespread enterprise adoption of multi-cloud and hybrid cloud for operational efficiencies has made flexible infrastructure the new standard. Organizations are bypassing single-vendor lock-in to spread workloads across multiple cloud partners. That means DevOps needs streamlined configuration and management tools that can track relationships between resources and keep track of compliance issues. Geopolitical considerations may enter the equation when cloud providers turn out to have close ties with governments antagonistic to other global trading partners. Monitoring instruments that provide observability across the range of vendors consuming enterprise data and applications is mission-critical.

    Resilience: Snafus happen. Platforms can automate continuous domain-level metrics to correlate performance from different sources (including logs, metrics, application tracing and outside penetration testing), speeding up detection and response time. Fast auditing and debugging tools, combined with real-time dashboard monitoring and system-wide alerts, plus an ounce of detection, equals a pound of cure. For agile DevOps teams, reactive firefighting is transformed into resilient firepower.

    Adopting an All-in-One Low-CodeOps Approach

    Low-code’s power to enable frequent iterations of software ideation, creation and deployment may seem contrary to the DevOps mission to minimize risks. However, the same radical abstraction and automation that has made software development faster, higher quality and more accessible to a larger cohort of developers can also reduce the operational burden of landscape management. Every low-code vendor touts the governance features in its platform. But the nuts and bolts of governance require some assembly, including system-wide actionable monitoring, testing alerts and compliance safeguards. Low-CodeOps combines the most efficient, agile, flexible development platform and powerful, streamlined operations tools, infusing both domains with solid guardrails for operational control and predictability. Adopting this approach will greatly increase time-to-value for enterprises, putting innovative digital solutions directly into the hands of end-users with the click of a mouse.

  • AIOps Builds Organizational Efficiency, Survey Finds

    AIOps Builds Organizational Efficiency, Survey Finds

    Enterprises entered 2020 with big plans. Post-pandemic, AIOps could help them achieve those goals, according to a new research survey from Digitate and CorporateLeader.

    Enterprises entered 2020 with the best-laid plans; many had efforts lined up to expand their use of cloud services and continue their steady efforts to digitally transform their businesses. Then the pandemic happened, and business and IT teams found themselves scattered and with a new playbook: make the organization as effective as it can be while working remotely.

    This sudden shift to remote work exacerbated existing challenges within organizations, from managing hybrid environments, personnel management challenges and teams finding themselves struggling to adjust. Meanwhile, enterprises still had to focus on staying ahead of their competitors technologically. All of these challenges meant there was no room for wasted or time-consuming efforts to manage the underlying business-technology systems.

    To help increase efficiencies, some contend that the answer is artificial intelligence for IT operations, or AIOps. Proponents of AIOps say that AI is needed to augment human decision-making today because of the complexity of modern enterprise environments. Today’s environments often find their operational data spread across on-premises and multiple cloud systems. The result is difficulty in managing infrastructure issues as they arise, and the considerable time it takes to identify and fix IT issues.

    According to a survey of leading European IT analysts and advisories, conducted in fall 2020 by operations automation software provider Digitate and executive network forum CorporateLeaders, the drudgery of managing routine tasks was the most commonly cited reason for turning to AIOps, which was followed by “no proactive capability of detection and correction of system issues” at 64% and “no flexibility to scale with business needs,” also cited by 64% of respondents.

    When respondents were asked to identify the key benefits of AIOps, they highlighted common strategic and operational issues that hinder performance. According to respondents, these include the removal of manual processes (91%), improvements in agility and reliability (82%) and reach predictability (73%). More swift problem resolution also was cited by 73% of respondents.

    In a case study shared within the CorporateLeaders report, Marion Fokkinga, technology advancement manager at Netherlands-based semiconductor maker NXP explained that NXP’s move to AIOps was driven by the company’s swift expansion. “Since 2015, we have been busy with multiple merger and acquisition projects,” Fokkinga explained. “As a result of this, there was less time for innovation and transformation in IT and in the business.”

    The company set out to automate routine operations tasks with AIOps so that their resources could be freed to focus on higher-value tasks. According to Fokkinga, AIOps enables tasks, such as report generation, that previously consumed half a workday to be completed in several minutes. Saving time is the most significant result, said Fokkinga. With AIOps, NXP reports a 97% improvement in incident response time and a 95% reduction of manual effort in some of its supply chain planning work.

    Of course, in the current treacherous business environment, there’s a lot more at stake than simply building enterprise operational efficiencies. Over time, it may come down to the very success of the organization. According to the survey, a full 82% of respondents said that AIOps is necessary to the future growth and transformation of their business. “Respondents who went on to give qualitative answers later in the survey explained that the complexity of hybrid and multi-cloud infrastructures, increasing transactional volumes and the criticality of online business systems make it impossible for traditional IT systems to manage every application and infrastructure without recourse to automation. It was highlighted that the overwhelming amount of data in today’s Operations are simply beyond human analysis capacity,” the report stated.

    One respondent said that, through AI, the performance of some tasks and decisions were improved, as routine tasks were eliminated. That has to be welcome news to enterprises struggling to remain competitive while simultaneously striving to remain productive and agile during a global pandemic.

  • Operations Model: DevOps, NoOps, AIOps or None of the Above Ops?

    Operations Model: DevOps, NoOps, AIOps or None of the Above Ops?

    Within an organization, an operations model must be fit-for-purpose and be able to adapt as business needs change. At the same time, changing technology necessitates an advancement in how an organization operates. Small and medium-sized enterprises are faced with the difficult choice of how much to invest in operations and which approach to choose. In light of the numerous possibilities, companies need to identify the right upgrades for their needs. For instance, should a company now using a traditional operations model invest significantly to reach DevOps? Or would AIOps better fit customers’ needs? Is NoOps an option?

    There is no “correct” answer. Each option comes with pros and cons and important budget considerations. That is why every business needs to evaluate the appropriate system for its products and its customers with three concepts in mind: cost, goals and experimentation.

    Operations Models

    Traditional IT operations (ITOps) is the process of designing, developing, deploying and maintaining the infrastructure and software components of a specific product or application. It also ensures the customer experience is delivered through traditional means of support such as tickets and escalated paths for resolutions.

    DevOps is a process for accelerating the delivery of features to the application or product infrastructure in a consistent model with less human intervention, while also providing a better quality product through automation in the software development life cycle.

    An AIOps process introduces data science into the operating model by learning the behavior of the systems, scaling according to the needs of a platform (both infrastructure and customer usage of the platform). It expands the horizon of DevOps through the introduction of machine learning, focusing on data generated from the hardware and software systems, and allows organizations to grow organically based on demand.

    NoOps, although the name suggests otherwise, is an advanced approach to managing IT operations through the mindset that everything is derived as development. This model is best applied with startups and companies with high technological maturity.

    Typically, these companies have advanced skill levels to deliver the best experience by adapting to the new industry trends of both software and hardware maintenance practices. NoOps is at the opposite end of the spectrum from traditional operations, and it’s largely a theoretical concept at this point.

    NoOps is a fully automated, cloud-based approach that eliminates the operations team, aside from an integrated team focused on life cycle. The self-managed system requires resiliency and a high level of maturity with fail points that have already been addressed. NoOps, or high maturity structures, will also provide quick learning on drawbacks, customer and product needs. However, moving toward a self-maintained system requires upkeep for advances and innovation.

    Invest in an Operations Model to Reach Higher Levels of Maturity

    The first factor to consider in evaluating an operational approach is cost. Each level of IT maturation is gradual, incremental and expensive. Costs vary depending on the size of the company and the type and number of products. A small company that seeks to move from a level 2 to 4 maturity would need to make a significant financial investment.

    Some questions to consider when looking at operations model upgrades include:

    • Does this make sense for the business?
    • Will the investment allow for scaling?
    • Will it increase customer loyalty and trust in the brand?

    Next, it’s important to consider overarching goals and brand identity. The appropriate level of maturity might depend on the company’s value proposition. If, in the company’s vision, technology is at the heart of its journey, the company needs to invest, at a minimum, in a DevOps operating model to have the competitive edge to grow the business and meet its needs. If the company’s vision is to retain existing business and gain customers through continued investment in innovation based on the trend analysis of their core business, then AIOps will have the best ROI. If the company’s vision, culture and mission are intertwined with technology, adapting to NoOps is the right move.

    Know the Product, Customer and When to Halt

    Still, it’s important not to overreach with an operations model that might not fit the customer. To make a sound decision for the organization, it is important to understand the model, the product, customer base and when to halt an operational model and invest in a new one. The leadership team needs to ask how adaptable its end users are before implementing changes. Take, for instance, the poor consumer response to replacing live customer service representatives with automated features. Some end users were not prepared to adapt to the massive change in the platform or service offerings this move entailed.

    Product-driven businesses often don’t see investing in new operations models as worthwhile. This mindset can result in continual churn that creates a backlog and increased operating costs for maintaining systems and services. Another important consideration in deciding which operating model meets the company’s needs comes through experimentation. That is because finding the right operating model for an enterprise can’t be done with a cookie-cutter approach. The point is to maintain the existing quality and productivity while investing in outcomes that will garner more trust and confidence in the company. If you do this right, the end user will appreciate the changes, and the company will outperform the competition.

    Allow for Operations Failure

    Enterprises should expect failures during experimentation with different models and plan to pressure-test an advancing model for the cracks that will emerge. It is important to invest in failure to understand why and how the model did not perform to expectations. Those failures provide valuable insight and important data, result in significant learning about products and customers and help you know what to improve or modify going forward.

    An operations model serves as the backbone for any business that wants to grow. Smart operations investments will ensure adaptability, competitive advantage and customer loyalty. But there’s no one-size-fits-all, easy-to-follow recipe for each enterprise. Whether the investment results in AIOps, NoOps, DevOps or traditional operations is unique to each organization and needs to be calculated with the brand, customer and end goal in mind.