Welcome to The Long View—where we peruse the news of the week and strip it to the essentials. Let’s work out what really matters.
This week: CSS-Tricks gets bought by DigitalOcean, Peloton pedals while its business burns, and Arm lays off up to 15% of staff. (more…)
It appears that, in many IT environments, observability is becoming too much of a good thing. A global survey of more than 315 IT executives, cloud application architects, DevOps professionals and site reliability engineers (SRE) conducted by Dimensional Research on behalf of log data management platform provider Era Software, found 96% are witnessing an explosion of log data.
The survey also found 79% of respondents reported the overall cost of observability and log management will skyrocket in 2022 if current tools don’t evolve.
More than three-quarters of respondents (78%) also noted attempts to manage volumes of log data have had mixed or unwanted results such as an inability to access data. Nearly two-thirds (65%) are also evaluating their observability options while the other 41% are considering it.
Stela Udovicic, senior vice president of marketing for Era Software, said the survey makes it clear that as organizations rely more on logs to analyze IT events, the cost of storing all that log data is becoming a significant challenge. Overall, the survey finds usage of observability tools and platforms has jumped by 180% as IT teams struggle to manage increasingly complex IT environments.
On the plus side, more business users are benefiting from log data. A full 83% of respondents report that business stakeholders outside of IT use insights from log data, with 96% reporting log data is being used to solve business problems.
Observability has always been a core tenet of DevOps best practices, but achieving it has always been a challenge. Monitoring tools are designed to consume predefined metrics to identify when a specific platform or application is performing within expectations. The metrics tracked generally focus on, for example, resource utilization. However, whenever there is an issue, it can still take days, sometimes weeks, to discover the root cause of an issue through what amounts to a process of elimination.
In contrast, observability combines metrics, logs and traces—a specialized form of logging—to instrument applications in a way that makes it simpler to troubleshot issues without relying solely on a limited set of metrics that have been predefined to monitor a specific process or function. DevOps teams can employ queries to interrogate data in a way that makes it easier to discover the root cause of an issue. Observability platforms correlate events in a way that makes it easier for analytics tools to identify anomalous behavior indicative of an IT issue.
The issue that arises is that IT teams need to find a way to store all the data generated by the logs they need to analyze. Each IT team can, of course, could reduce the amount of log data they retain. However, there is always a concern that in the event of an incident critical log data won’t be available.
Udovicic said most IT organizations have historically viewed storing log data as a necessary evil. The primary issue most of them encounter is the simple fact that log data is not easy to work with, she noted.
Regardless of how organizations view log data, the amount of it that needs to be managed will only increase as DevOps workflows become more mature. The issue IT teams need to come to terms with now is finding a way to minimize the cost of storing it all.
Measuring and mitigating the security risks in open source software is becoming a major issue in the software development community. Attacks on open source software (OSS) are on the rise; open source supply chain attacks grew 650% in 2021—a staggering number.
2021 and early 2022 saw significant attacks which impacted more than 1,000 organizations. The most prominent recent OSS attack was the Log4Shell vulnerability disclosed in December 2021. Attacks on the OSS supply chain and on OSS impacted nearly every single software organization on the planet. Everybody is scrambling to continuously patch systems as new vulnerabilities continue pouring in.
To properly navigate this environment and keep our systems safe, we as a community need to understand the security risks of OSS and be better prepared to handle them, collectively and individually.
There are a number of risks that make OSS more vulnerable without proper security measures.
To start, we must recognize that open source software may be free to download and require no payment, but it comes with a massive security cost.
A significant portion of open source software is developed by volunteers in their free time. Most of the prominent open source volunteers actually have full-time jobs, and the code they contribute to projects like the Log4j logging libraries are written on nights and weekends; on their own time and not on the company dime. Not surprisingly, security isn’t high on their priority list. They don’t have the resources to invest in a secure development life cycle or have the time and bandwidth to maintain their code. This may mean that security vulnerabilities remain unaddressed for long periods of time, sometimes indefinitely.
In software development, Linus’s law states “…given enough eyeballs, all bugs are shallow.” However, this is only true if those eyes are looking in the right place. In the case of open source, all code is open but there aren’t enough eyes paying attention to it to spot all the bugs. In addition, organizations may not consider the security consequences of their decision to consume a particular piece of open source software and adding it as a dependency. Dependency trees become more and more complex; harder and harder to monitor and secure, as you keep adding more third-party components to a core codebase. Each new dependency might bring additional nested dependencies—perhaps hundreds or thousands, depending on the size of the library added or the role it plays in the application architecture. For example, the Kubernetes project—one of the most successful and widely adopted OSS projects—has several thousand dependencies, many of which have additional secondary and tertiary downstream dependencies.
The beauty of the open source ecosystem is that there is a solution for everything. The problem with the open source ecosystem is that there’s not just one solution, but several dozen. For example, if you need a JSON parser for your application, there are several to choose from. If your application is running a critical service or infrastructure, then picking the right JSON parser—one that is secure and comes from a responsible project or maintainer—becomes very important. A JSON parser will serve as a critical component for your application, so it will impact the security of your entire system. If you chose the wrong JSON parser, you might pay a big price down the road in security risk and subsequent breaches.
One potential solution to enhance OSS code security is the creation and implementation of a security risk assessment gate as part of your OSS ingestion pipeline. Putting this process in place will help you understand whether consuming a certain piece of software is high risk and avoid software that does not meet your risk criteria.
Here is an example of how this might work. Consider five different JSON parser libraries in GitHub, all of which have a high number of stars and appear to be highly rated. How can we assess the risk of these libraries and quickly and easily eliminate ones that are too risky? One of these JSON libraries is hosted under a user account. Most of the commits are done by a single developer without reviews. This alone indicates that using this library in a critical system presents a high risk. This lone developer could commit malicious code without review. The maintainer’s token access could be leaked, allowing attackers to insert malicious code. The developer might also turn over committer control of their library to a malicious actor without notifying users (who might not pay attention to the notification anyway). Identifying this risk required no more than a few minutes of review of the repository and cost little in time or effort.
More broadly, we need to understand the full security posture of an organization or developer and their open source code prior to adding it to critical systems. Because OSS is a complex ecosystem and many pieces of OSS are complex, the risks they generate are similarly complex. Our traditional, myopic view of these risks is that we just need to find all the security vulnerabilities in our codebase and fix them. This does not resolve all risks and may lull us into a false sense of security.
We need to understand the full comprehensive security posture of an open source project. That involves understanding a broad range of risks beyond code vulnerabilities. Here are some of the key risks to consider beyond ensuring that your codebase is well-secured and patched.
Supply Chain Attacks on the Build Pipeline
Attacks like SolarWinds and CodeCove taught us that supply chain risk is a very big deal and that tools are just as much at risk as code. Attackers have found it much easier and much more reliable to attack build pipelines, rather than finding vulnerabilities in code.
Security Risks Revealed by Continuous Security Testing
Continuous security testing is critical to maintaining the security of a codebase. There might not be code vulnerabilities today, but as developers continuously modify code and iterate on applications, vulnerabilities may be introduced. This can mean testing code as part of the CI/CD process with every new code push. For example, we have found numerous critical bugs in OpenSSL libraries. All were caught on a master or main branch and never made it to a production release.
Project Maintenance and Capabilities
One of the really big risks with open source is the quality and frequency of project maintenance. Issues raised may not be addressed in a timely fashion. Maintainers of a project may have a long backlog of issues, indicating that they are not actively maintaining a project. It is not uncommon for critical open source projects used in millions of applications and comprising a significant dependency to the global software supply chain to not have been updated in a year or more. In addition, maintainers of OSS projects may not know how to respond to security incidents and fix security vulnerabilities. This is particularly true if the maintainers do not work at a large organization, do not have cybersecurity or secure development experience or have never suffered through the toil and trauma of addressing a major vulnerability in their code on short notice.
A number of efforts are underway to create free tools to help developers and application security teams better measure and address supply chain risk programmatically and analytically.
As a tool to help organizations better address software supply chain risks, Google created a novel framework that we call Supply Chain Level for Software Artifacts (SLSA). The project has been underway at Google for more than a decade and has been open sourced as part of the OSSF. Contributors include VMware, Citibank and Datadog, among others. Google hopes to standardize SLSA and make it an industry benchmark for evaluating supply chain risk. Right now, SLSA is focused primarily on build process risks but the hope is to expand it in the near future to other areas, such as vulnerability and maintainer risks.
In order to scale the process of assessing, identifying and mitigating risks in OSS, we need robust automation. In 2020, as part of the OSSF, Google released a new project called Open Source Security Scorecard. The project aimed to automate the analysis and trust aspect when evaluating the security posture of open source code. Further, the Scorecards can provide specific security recommendations. The Scorecard process can incorporate many other automated tools, including checking against vulnerabilities using the Open Source Vulnerability (OSV) database and running dependency checks using open source dependency trackers like Dependabot. The scorecards project also makes it easier to report new vulnerabilities and get those to the right people in a project.
No one denies that OSS has become crucial to the vast majority of applications running the world’s technology foundation. The span of OSS continues to grow, as more and more languages and frameworks emerge and specialization increases. No one wants to reinvent the wheel and rewrite fresh code to perform the same tasks with every new application. The model of reuse, dependency and distributed software development works and speeds innovation. That said, the open source community and the supply chain used to build OSS need a more rigorous approach to reducing risk. Given the scope of the problem, automation and standardization are helping to scale toward the stated goal. We are in the early stages of creating the tools and infrastructure to automate the risk identification, measurement, assessment and mitigation processes. Security frameworks like SLSA and new processes like entry gating new OSS code with security scorecards will direct organizations down that path and move us toward a more secure open source future.
About Secure Software Summit
The Secure Software Summit brought together the world’s leading innovators, practitioners and academics of secure software development to share and teach the latest methods and breakthroughs on secure coding and deployment practices. If you are about developing, releasing and securing software, delivering new features fast and building things right from the start, click on the link below to get access to all of the sessions from this summit.
https://go.shiftleft.io/secure-software-summit-2022-replay
Last year, the Linux kernel turned 30. If you’re someone who’s been immersed in the Linux world since Y2K like I am, it may feel a bit surreal that so much time has passed since the kernel’s inception.
As a training architect at A Cloud Guru (ACG), I teach courses about all things Linux and specialize in hands-on lab-based learning. Before joining ACG, I worked as a Unix systems engineer at GE and IBM as well as a technical account manager and customer advocate for Red Hat. I’m hugely passionate about Linux, just as so many other engineers and enthusiasts are, not just because of its importance to our careers but its impact overall.
In its 30 years of existence, the Linux kernel has had a massive impact on the modern computing landscape—revolutionizing what’s possible for operating systems and allowing countless tinkerers to get their hands dirty in the process. Linux has also become the foundation of paradigm-shifting innovations over the years due to its ever-evolving nature.
Here are some of the key reasons why the kernel is so valuable and why it will continue to be a major player in the computing landscape for years to come.
My first exposure to Linux was in college in 1999. I took an introductory Unix Shell Scripting class and Fedora was installed on the lab servers because it was free and easily scalable. I view Linux as my entry point into customizable operating systems. After learning basic Unix commands on Linux, I ultimately went on to pursue a career supporting multiple commercial Unix vendors.
As Linux made its way into larger enterprise-level companies, I quickly returned to working with Linux distributions. Something I’ve always loved about Linux is the capacity to fine-tune your system to support the applications and distros that work best for your projects. Linux improves the functionality of whatever applications you’re running. The Linux kernel has literally changed how the world processes information, which is why I’m so invested in the software.
Now, it’s my job to share my knowledge and passion for Linux with other technologists. At ACG, I develop courses to help aspiring Linux experts learn how to optimize their systems. Additionally, I contribute to Linux this Month, an ACG-hosted web series that provides monthly updates from the global Linux community.
The fact that I am able to build a career around teaching Linux and staying up-to-date with Linux news shows the vast use cases and applications of the kernel. What makes Linux so unique and evergreen is its open source nature—Linux innovations are only limited by the creativity of the technologists who use and adapt it.
When Linux first arrived, it was mostly a hobby for enthusiastic engineers and computer science students who could contribute by developing code. The steep learning curve associated with fitting Linux to your machine was a barrier to entry for more novice programmers.
Over time, this has changed considerably. Online forums, workshops and classes have made Linux more accessible to the average internet user. The free sharing of ideas has come to epitomize the open-source community, and for software engineers, Linux is at the heart of this community. This democratization of Linux has had incredibly positive impacts on the computing world.
Now, Linux is everywhere. Enterprise-level companies use Linux distributions to process the biggest production workloads in the world. It has replaced proprietary commercial Unix operating systems in very large companies with better stability and less downtime. Because Linux systems can be as small or as large as you want, it’s also now being used in our homes for smart and mobile devices as well.
The open source nature of Linux is incredibly beneficial for these enterprise-level companies. Linux distributors leverage the contributions of the entire open source community. This wide range of contributors produces a more stable product with more features but also ensures that the OS keeps growing and solving real-world problems that are beneficial to a wide range of users.
Potentially the most impactful outcome of the kernel is its importance to the infrastructure of modern cloud computing. Linux’s scalability has paved the way for supercomputers and server farms to function efficiently while requiring relatively lightweight computing resources. In fact, Linux supports about 90% of the public cloud workload.
Without Linux, the cloud as we know it would not exist. This is, in part, because Linux has become so ubiquitous—its use cases are nearly limitless. Because it has been time-tested, many engineers and IT professionals have a solid grasp of Linux fundamentals, making it an attractive choice for enterprise companies dealing with the cloud.
Cloud-based software and products are increasingly becoming the norm in the engineering world. Unsurprisingly, major cloud providers such as AWS, Azure and Google Cloud are all supported by Linux as well. Linux is unique because it is a shape-shifter that can conform to the needs of any given engineering environment, and it’s incredibly stable because of the army of contributors that fortify weak points in the software.
The concept of Linux is just as important as the Linux product. The concept allows a free and open source operating system to be refined, reinforced and replicated across an endless web of contributors. Thirty years is a long time for software to be relevant, especially with the ever-shortening tech product cycles. Because it was designed with the intent to be changed and updated by an open source community, Linux has no foreseeable expiration date.
Had Linux not achieved the prominence it has today, we would see more commercial Unix vendors attempting to solve some of the problems that Linux addresses, but none would address them all. Additionally, customers would have to choose which OS to invest in based on which addresses some of their use cases, but none would be as beneficial as the Linux OS.
Linux is always growing and will become even more popular within the next few years. As more people become familiar with Linux and learn to use it, I see major potential for growth in the mobile computing space, within personal computers, and across small and large companies. In fact, we are already seeing it filter into home gaming systems and Raspberry Pi projects. With Linux, the sky’s the limit!
The rapid pace of modern software delivery and increasing fleet sizes have transformed how organizations see and handle security strategies. Compliance-as-code is required for today’s organizations that need security as a fundamental part of business processes. It is no longer possible to manually manage compliance with dedicated security teams.
Balancing security with growing infrastructure needs means IT security and compliance are non-negotiable. There is no room for more uncomfortable trade-offs between risk and an organization’s ability to deliver market-ready solutions quickly and efficiently.
Complying with growing security and compliance regulations in today’s world of rapid innovation is a constant challenge that affects all organizations, large and small. Organizations are implementing automated solutions to eliminate reliance on traditional, slow, error-prone manual processes because of tighter regulations in the industry and greater risks associated with security attacks and compliance violations.
Compliance-as-code is the codification of compliance controls to automate their adherence, application and remediation. It includes the tools and practices that enable DevOps and developer teams to incorporate the three key compliance activities:
There are many important use cases for managing the compliance of endpoint state as-code including confidential data protection, detecting shadow IT resources, network security, data exposed to public access and code licensing compliance.
Most organizations still struggle to stay secure and comply with regulatory standards. Many organizations lack visibility across heterogeneous infrastructure and applications. They also possess inconsistent language for communicating requirements between development, security and operations (dev, sec and ops) teams and are unable to remediate findings.
Compliance-as-code enforces a comprehensive compliance strategy that involves a robust set of controls; for example, managing data storage locations and access control management.
Ensuring that these are followed at scale is critical to the success of a digital business as more organizations shift to the cloud, there are more possibilities of non-compliance. Reports suggest that a regulatory compliance violation can cost businesses $15 million on average.
This violation is avoidable if DevOps, development and DevSecOps teams can automate compliance by adopting a different mindset and writing understandable code. The adoption of the “as-code” strategy led to infrastructure-as-code (IaC) which is one of the success factors in DevOps teams. Audits are streamlined and allow individuals to focus on higher-value activities.
Managing a fleet through compliance-as-code is not without its challenges. First, teams must bridge the talent gap as endpoint compliance is typically managed by IT admins, who are not inherently comfortable with an as-code approach.
Typically, compliance requirements are dense and difficult to comprehend by other stakeholders. Understanding the compliance requirements of the organization and collaborating with other stakeholders to convert them to as-code is the biggest hurdle to getting started.
Developer teams must find the right set of tools and integrate them into DevOps and DevSecOps workflows. This is not just a process change but a culture change. A boil the ocean approach will not just be painful but success will be close to impossible. Taking small steps and continuously improving is the mindset one has to adopt.
Whether you are talking about end-user devices like desktops, laptops or mobile devices or non-server edge devices like point of sale (POS) or IoT devices, endpoint devices are extremely vulnerable to exploitation from bad actors. Thus, it is necessary for organizations to bring them under the organization’s security policy and governance requirements just like server-side devices.
There are a number of advantages to adopting an endpoint compliance-as-code approach:
Managing diverse IT infrastructure fleets through compliance-as-code ensures that endpoint devices meet a variety of security standards. The IT team is empowered to create rules that can enforce compliance and establish security baseline standards within the organization. Compliance checks are validated and more visible at each stage of the software development life cycle.
This guarantees detailed visibility and thorough compliance standards throughout the IT ecosystem.
With the increase of supply chain attacks on everything from logging software like Log4j to takeovers of important JavaScript packages to compromises of network utility tools like SolarWinds, more and more organizations are recognizing the need to adopt a zero-trust mindset. Zero-trust can improve security, reduce risks and give organizations greater confidence in the integrity of their IT infrastructure and applications. To correctly set up a zero-trust architecture, you need to understand what it actually takes to make systems zero-trust.
Due to the rise of interest in zero-trust, many misconceptions have built up that bear correcting.
So, what is zero-trust? Zero-trust is a framework and approach to security that assumes an organization’s IT assets — software, networks, data — are constantly at risk and must be secured through continuous verification of the identity, and the application of least privilege principles against any identity — human or machine — operating inside the IT environment. The core of zero-trust is the lack of inherent trust for any identity as a managing approach to building more resilient applications and segmented risk.
Zero-trust has several key guiding principles.
The conceptual language of zero-trust makes it sounds like an identity architecture and network security approach, floating above developers. For building and deploying secure applications, developers and DevOps teams must also adopt a zero-trust mindset. In fact, there are clear synergies between zero-trust and application security — and particularly with regard to open source software (OSS). OSS developers may not spend sufficient time and energy on securing their code because it is not what they like about writing software. For this reason, if you are including OSS code (and nested OSS libraries) in your applications and environments, then you have even more reason to apply zero-trust frameworks.
A useful way to think about this is to consider zero-trust as applicable from the inside out, from the outside in and to everything in between linking internal and external, applications and APIs, code and integration/deployment pipelines and partner environments. Properly executed, zero-trust allows your developers to go faster but also be safer. Additionally, it is useful to view zero-trust through the confidentiality, integrity, availability (CIA) lens. You should design your zero-trust architecture to deliver these elements in a priority hierarchy that maps to your business needs. There are a few different ways to implement zero-trust principles in the software supply chain.
Conduct a software asset inventory, looking at and categorizing all code running in your environment. This makes it easier for you to identify unauthorized changes and build more resilient applications. Set up mechanisms to detect unauthorized changes or anomalous software behaviors. You may want to apply these guidelines to close partners and their software and supply chain, as well. Remember that attacks may be from you, through you, or at you.
Analyze and categorize user entitlements to set a baseline for least privilege. Conduct an inventory of all identities active in your environment to identify known identities (and quickly recognize unknown identities). Set up mechanisms to analyze the behavior of identities to spot anomalies and create a baseline for context and behavioral privilege and access policy decisions. Include partner identities in this inventory: Your greatest risk may come from unsecured partners that attackers use to gain access to your environment. The inventory should also include service and admin accounts. The inclination is to set their privileges and forget about them, often awarding broad access so as not to slow them down and also potentially providing an unmonitored backdoor to attackers.
Inventory your external and internal supply chain. This includes all dependencies in the code, services called, APIs exposed or accessed and tooling for developing, hosting, integrating and building applications. For example, you may want to analyze your CI/CD pipeline and deployment environments through the lens of segmentation and least privilege to limit access depending on the pipeline stage. Put in place measures to segment the software environment and prevent lateral movement, like application sandboxing (but make sure to also tightly secure your sandboxes).
Developers need to apply threat models to their code and adopt the mindset of considering what an attacker might access and what they might do should they compromise an application or the surrounding supply chain. Obviously, for all code — and especially OSS code – developers will need to continuously look for vulnerabilities and scan code using tools like static code analysis and dynamic application security testing. Zero-trust means continuous verification of your own code, not just the entities using the code.
As we continue towards a future of more complex and integrated application environments and more usage of OSS code for core application functionality, the zero-trust future becomes zero-trust forever. It will simply become a way of life in application development. That said, we should keep in mind four key guiding principles.
Know Your Current State—Be aware of where your environment stands and your security posture, and direct and indirect impacts of security vulnerabilities.
Assume Breach—Be able to identify anomalous behavior and continuously consider what could be done by an attacker if your app was used maliciously?
Assess—Continuously assess the status and changes in the environment. Ask questions like: Who has what, when and where? Where do your integrated partners and vendors stand in terms of security?
Continuous Improvement—Periodically assess and compare your zero-trust efforts against desired security posture and make improvements that align with objectives around confidentiality, integrity, and availability. Constantly guard against risk.
These four principles map to five key takeaways that any developer, application security team, DevSecOps team or platform team can apply today and forever.
Following these guiding principles and basic framings will ensure your zero-trust journey creates strong defenses that help protect your development supply chain from the outside in, inside out, and with integrated partners.
The Secure Software Summit brought together the world’s leading innovators, practitioners and academics of secure software development to share and teach the latest methods and breakthroughs on secure coding and deployment practices. If you are about developing, releasing and securing software, delivering new features fast and building things right from the start, click on the link below to get access to all of the sessions from this summit.
DevOps has been with us long enough now that there is a lot of legacy DevOps stuff running in all of the places I have eyes into. Ignoring items that are not actually working; those pieces that pose a serious security threat because they house vulnerabilities that have been fixed—there is a lot of “It just works,” piping in place and even a bit of “We’ve always done it that way.”
Considering DevOps is still often pushed as “the inevitable new hotness” by marketers, I find this interesting; the fact that DevOps is now the de facto standard across enterprises is a topic for another day. I want to talk about the stuff running quietly, relieving you of the need or desire to think about it while focusing on issues that do need you to think about them.
But that type of thing is exactly what created the need for DevOps in the first place. DevOps came in like a wrecking ball and forced you to look at the unnecessary steps that had built up around the items in the organization/culture/infrastructure, often around these very “This is just how we do it,” or “That ain’t broke, let’s look at this other thing, instead,” attitudes.
At Ingrained Tech, we have scripts that run our backups to the cloud. They are part of a fully automated process; we designed them to make sure our system of what gets backed up and when highly adaptable. The system runs like a champ every night unless we tell it not to.
Our cloud storage provider has come a long way since we first used their API to get the data up there and wrote those scripts. In fact, customers like us are the reason the provider keeps backward compatibility with those APIs. Which means it is time for us to revisit, use the newest APIs and see which of the many, many new features we want to take advantage of. “Good enough” is just that. The system isn’t broken—we get viable backups out of it (which we needed not too many months ago, so we know we get good backups), but we’re not getting maximum value out of it; we had to do a bit of hoop-jumping to get the data set back up the way we wanted.
Our simple backup routines are nothing compared to similar “good enough” arrangements I’ve seen in larger companies. “Good enough” arrangements that are creating larger and larger speed bumps that sometimes resemble the exact environment that made the huge gains of early DevOps implementations inevitable.
Take a look at your infrastructure and toolchains with a critical eye. Look for low-hanging fruit, and knock out some of the baggage that has built up. Schedule some process renewal days, where a team member or two specifically looks at everything in the environment with a critical eye, and see what they propose.
And keep rocking it. Speed bumps or not, you’re making the apps and infrastructure that keeps the entire company running. Remember that, and keep a smile on your face while you’re solving recurring problems.
A recent cloud observability survey found nearly two-thirds (64%) of respondents reported the adoption of public clouds has made it more difficult to achieve observability. Nearly half of respondents (48%) said they now consider providing real-time insights into application/infrastructure environments to be a top priority.
The survey polled 357 IT professionals and was conducted by Enterprise Strategy Group (ESG) on behalf of Yotascale, a provider of cost optimization tools for cloud computing environments.
Yotascale CEO Asim Razzaq said that attaining and maintaining observability becomes more challenging with each new cloud platform adopted. The survey found that 71% of organizations now rely on more than three cloud infrastructure providers.
A full 84% of survey respondents also noted that tracking costs and cloud cost allocation is burdensome and time-consuming, and 61% admitted they lacked sufficient visibility into which teams own what cloud resources. Nearly a quarter (24%) are considering making changes to their monitoring and observability practices as infrastructure costs rise.
Observability has always been a core DevOps tenet, but achieving and maintaining it is challenging. Most DevOps teams today aspire to be able to maintain some level of continuous monitoring. However, as it becomes easier and less costly to instrument applications, interest is rising in observability platforms that simplify the investigation of anomalies indicative of an issue that could disrupt an application environment. As that shift occurs, more organizations will seek to correlate events with costs as part of an effort to rein in the costs of IT, said Razzaq.
In the immediate aftermath of the COVID-19 pandemic, the number of workloads being deployed on public clouds increased sharply. In theory, many organizations signed enterprise licenses that reduce cloud costs if they consistently consume a certain level of infrastructure resources. In practice, however, individual developer teams are showing a preference for one cloud platform over another based on the attributes of their workloads. As a result, however, it becomes more challenging to not only consistently observe diverse platforms but also to control costs.
It’s still early days as far as the adoption of observability platforms is concerned, but it’s apparent that as application environments become more complex the plethora of monitoring tools that DevOps teams rely on will need to be streamlined. It’s not uncommon for DevOps teams to have multiple monitoring tools that surface a set of pre-defined but often conflicting metrics. Observability platforms unify logs, metrics and traces in a way that makes it simpler to launch queries to identify the root cause of an issue.
The rate at which DevOps teams will embrace observability will naturally vary. In 2022, the biggest obstacle might not be the technology as much as it is simply understanding what queries can help DevOps teams better understand the root cause of an IT issue before there is a major disruption. In the long term, it’s expected that machine learning algorithms will leverage the data collected by observability platforms to automatically identify issues that might lead to a disruption long before it actually occurs. Regardless of approach, however, the ability to manage complex IT environments in the months and years ahead will continue to steadily improve.
CircleCI today published a report that found most successful software engineering teams have average workflow durations of five to 10 minutes, with any failed run being fixed or reverted in under an hour.
The report was based on two years of data gathered from more than a quarter-billion workflows and collected from nearly 50,000 organizations using its continuous integration/continuous delivery (CI/CD) platform. The CircleCI 2022 State of Software Delivery Report also found those teams have success rates that are above 90% for the default branch of their application and deploy at least more than once per day.
Mike Stahnke, vice president of platform for CircleCI, said the report also makes it clear there is no correlation between team size and the level of DevOps success achieved.
Organizations that focus on optimizing pipelines are also better able to integrate testing within their DevOps workflows, he noted.
In general, Stahnke said more DevOps teams are relying on cloud-based CI/CD platforms versus opting to deploy and maintain infrastructure themselves. That shift also lowers the barrier to entry for organizations looking to embrace DevOps best practices to accelerate application development, he added.
Less clear is the degree to which organizations have embraced both CD and CI. While many developers routinely employ CI to accelerate application development, CD has proven to be a more elusive goal. Each platform a DevOps team supports tends to have unique attributes that require them to rely on manual processes to deploy applications. However, as DevOps processes continue to mature, Stahnke said the CI and CD integration will become more seamless. In fact, as the number of applications that organizations build simultaneously increases, the pressure to automate CD will also rise.
However, it remains to be seen to what percentage of organizations will opt for a tightly integrated CI/CD platform versus separate CI platforms for developers and CD platforms for IT operations teams that are more loosely coupled.
In the meantime, organizations of all sizes are clearly more focused than ever on developer productivity. In the wake of the downturn brought on by the COVID-19 pandemic, there was a rush to launch a range of digital business transformation initiatives. The issue that many organizations are now encountering is they need to find ways to enable a finite number of developers to simultaneously build and deploy multiple applications with multiple interdependencies. Many organizations are trying to address that challenge by sharing best practices across DevOps teams that have varying levels of maturity. Few organizations have a set of standardized DevOps practices that are consistently employed across every application development team within an enterprise.
It will be up to each organization to decide whether to compare their level of DevOps expertise to other organizations. Each application development team faces a unique set of challenges. The most important goal is simply to continuously improve at a time when demand for custom applications has never been greater.
It seems SREs are of two minds when it comes to AIOps. On one hand, AIOps’ potential is pretty exciting. By automating complex workflows and troubleshooting processes, AIOps could make your life as an SRE much easier.
But on the other hand, some SREs may choose to view AIOps with disdain and distrust. They might think of AIOps as just another trendy buzzword that doesn’t live up to the hype and that can become a distraction from the SRE tools that really matter.
Which perspective is right? Should SREs embrace AIOps with open arms, or should they resist marketers’ efforts to position AIOps as the latest, greatest tooling innovation in the IT industry?
Those are subjective questions that we can’t answer definitively, but let’s at least gain some perspective by examining what AIOps means for SREs.
If you keep up with IT buzzwords, you’ve probably at least heard the term AIOps—which is short for artificial intelligence for IT operations—and that it refers to the use of AI and machine learning (ML) to help automate IT operations workflows.
The big idea behind AIOps is that, by using AI and ML to perform advanced analysis of large volumes of data from IT systems, IT and SRE teams can solve complex problems more efficiently than they could using a traditional, manual approach.
AIOps can, for example, help to surface the root cause of a performance issue in a complex, multilayered environment like Kubernetes. Or it could make recommendations about how best to resolve an incident.
AIOps entered the IT lexicon in 2016, when Gartner coined the term. Now, six years later, it’s a relatively well-established tool domain.
Despite the fact that AIOps has been around for some time, it doesn’t yet appear that many SREs have bought into the AIOps revolution. Catchpoint found in a 2021 survey that just 7.5% of SREs reported that AIOps tools delivered high value to their organizations.
It’s unclear exactly why there are such low rates of excitement about AIOps, but we’d speculate that there are a few key factors at play:
From an SRE’s perspective, then, AIOps may appear over-hyped, overly complicated and underperforming compared to traditional approaches to SRE.
SREs’ wariness toward AIOps is valid—but only to a point. It’s important not to let suspicions about the limitations of AIOps turn into excuses not to use AIOps at all. AIOps has some value to offer to SREs, even if it’s not perfect.
For example, AIOps can play a role in reducing toil. To the extent that AIOps tools can recognize complex patterns or discover interrelated data sets more quickly than human engineers, AIOps reduces the time SREs have to spend manually troubleshooting problems or poring over complicated information.
AIOps also helps to enable a more proactive approach to monitoring and incident management. If AIOps tools can alert SREs to emerging issues before they would otherwise recognize them, AIOps can help the SREs to get in front of the problems before they turn into true incidents. That’s better for SREs and end users alike.
There is also an argument to be made that AIOps can help SREs do more with fewer engineering resources. If you can use AI to automate some aspects of monitoring and incident response, you can maintain the same levels of availability and performance with fewer human engineers on hand.
None of the above is to say that AIOps can replace SREs, or that it magically solves every problem SREs face. Anyone who believes AIOps is a silver bullet has bought into the marketing hype to an unhealthy degree.
Nonetheless, AIOps tools do offer value to SREs by making their jobs easier in some respects and improving reliability outcomes.
So, while it’s wise to maintain a healthy perspective about the limitations of AIOps, SREs shouldn’t rule out AIOps tools as one way to improve reliability engineering.