devopsy.kinsta.cloud

Category: Continuous Delivery

Continuous delivery is a software engineering discipline that allows organizations to consistently develop and release quality software at a rapid pace. It is achieved by automating the software development lifecycle, enabling tasks such as development, testing and release to occur concurrently and continuously.

Survey Surfaces Lack of Significant Observability Progress

A global survey of 500 IT professionals suggests organizations are not making a lot of progress in their ability to truly observe application environments, especially as they become more complex with each passing day.

The survey, conducted by Logz.io, a provider of an observability platform, found only one in 10 respondents reported they have full observability into their application environments.

Logz.io CTO Asaf Yigal said while more DevOps teams than ever are collecting logs, metrics and traces, most of them have yet to determine how to operationalize all the data being collected. As data is ingested, it doesn’t just need to be stored; it also needs to be correlated to the various services that make up an application, he noted.

The more organizations deploy cloud-native applications in production environments the more pressing this issue becomes. All the microservices that make up those applications are now generating a massive amount of telemetry data that results in more alerts than ever being generated.

Not surprisingly, the biggest challenges organizations encounter when managing Kubernetes clusters in production environments are monitoring/troubleshooting (40%), followed closely by security (37%) and networking (33%).

Many organizations simply don’t have the skills required to manage cloud-native applications. Nearly half of respondents (48%) specifically cited a lack of knowledge as the biggest challenge they encountered when trying to observe these types of applications. From an IT management perspective, most microservices all use the same basic template, so it’s difficult for DevOps teams to identify which microservices are likely to have the biggest impact on service level objectives (SLOs) and service level agreements (SLAs) in the event of a disruption, noted Yigal.

In the absence of the ability to determine the actual root cause of an issue, alert storms conspire to increase fatigue—and that eventually results in higher levels of burnout across the DevOps team, noted Yigal. In fact, 82% of respondents said their mean-time-to-resolution (MTTR) during production incidents was more than an hour.

The survey also noted that more than half of respondents (52%) worked for organizations that are simultaneously trying to rein in monitoring costs. More than three-quarters (76%) of respondents also noted that open source OpenTelemetry (OTEL) or OTEL-centric tooling was at least somewhat important to their overall observability strategy.

In addition, 87% of respondents said their organization is already using some form of platform engineering to manage DevOps workflows at scale.

It’s not clear how quickly organizations are adopting observability tools and platforms, but too many DevOps teams lack the visibility needed to identify the root cause of an issue. As a result, variations of the same problems often keep manifesting themselves because previous remediation efforts simply didn’t go deep enough to resolve the core issue.

Of course, there may come a day when machine learning algorithms, along with other forms of artificial intelligence (AI), will make it simpler to surface these issues. The challenge, in the meantime, is laying down the observability foundation today to provide access to the data that will be required to train those AI models.

March 27, 2024
Oracle Looks to Increase Pace of Java Innovation

With the latest release of Java, it should become simpler for DevOps teams that build and deploy applications with the most widely used programming language in the enterprise to innovate faster.

Georges Saab, senior vice president, Oracle Java Platform, said Java 22 is unique in the history of Java in that it is specifically designed to enable developers to invoke capabilities that are available in beta and preview while still working with the other functions supported in the venerable programming language. The overall goal is to increase the pace of innovation in environments based on Java applications, he added.

The latest edition of the Java Development Kit (JDK), for example, provides access in preview to capabilities developed under a Project Amber initiative that makes it simpler to express strings that include values computed at runtime while also improving the security of programs that compose strings from user-provided values and pass them to other systems.

Meanwhile, a Project Loom initiative, also available in preview, streamlines error handling and cancellation in a way that improves observability using an application programming interface (API) that enables structured concurrency, while a Scoped Values capability provides a way to share immutable data within and across threads.

A Project Panama initiative adds a Vector API that promises to compile instructions at runtime when, for example, making use of retrieval-augmented generation (RAG) techniques to extend artificial intelligence (AI) models.

In terms of additional capabilities that are now generally supported, Java 22 adds an API to enable Java programs to safely interoperate with code and data running in memory outside of the Java runtime.

Most importantly, from an application performance perspective, the latest iteration of the language streamlines garbage collection processes in memory to reduce the number of times a Java application might otherwise need to be paused.

Despite the number of alternative programming languages that exist, the bulk of enterprise-class applications continue to be built in Java. While many developers are at least familiar with multiple programming languages, most tend to favor one primary programming language. Java developers, especially, have shown little inclination to abandon the language even as other programming languages, such as JavaScript, have emerged that are either simpler to use or, as in the case of Rust, may provide more robust security capabilities.

Less clear is at what rate developers are upgrading to the latest iteration. In some cases, developers are still making use of version 8, 11 or, more recently, 17. It may be a while before the bulk of Java developers are ready to make the jump to Java 22. In addition, DevOps teams will then need to decide when there is enough critical mass to warrant upgrading runtime environments to support another iteration of Java.

One way or another, either via an edition of Java provided by Oracle or another vendor, the programming language isn’t likely to be supplanted any time soon. The challenge now is providing support not just for the various versions of Java that might be running but, inevitably, all the other software artifacts written in various languages that are now also routinely moving through DevOps pipelines.

March 26, 2024
From Chaos to Clarity: Streamlining DevSecOps in the Digital Era

These days, securing the software development lifecycle from end to end requires organizations to deploy, maintain, and master a cacophonous mix of tools that often do more to create discord rather than bring harmony to DevSecOps processes. The reason for this is simple: Each tool employed to secure a software supply chain independently runs scans and generates alerts that lack context, are often redundant, or flat-out contradict one another.

Developers and the software engineers that support these applications are, of course, expected to correlate the steady streams of alerts and address vulnerabilities to meet security thresholds required to propagate the release to an upstream environment. The reality is that not all of the vulnerabilities flagged as high and critical need fixing, nor is it possible for a team to address all of the vulnerabilities. On average, a development team has the bandwidth to address 10% of its vulnerability backlog in any given month. This makes it imperative to prioritize the vulnerability backlog based on impact rather than severity if we are to give these development teams a ghost of a chance to succeed in improving their security posture.

The constant alert storms also lead to fatigue and introduce a real risk of false negatives. The one that got away!

DevSecOps workflows today are little more than sections of an orchestra playing without a score with the expectation that some methodology to consistently lock down a software supply chain will magically manifest.

What’s obviously missing is an orchestration framework that triggers the security workflow in response to changes within the SDLC. Ideally, it works seamlessly with the CI/CD workflows, but without being dependent on them, creating a symphony capable of providing a better security posture for the organization.

The Trouble With DevSecOps

In cloud-native, microservices-based application development environments, teams could be developing using multiple technologies, each optimized for delivering a certain service. Each technology may require a certain specialized scanning tool—and that’s just code; by the time we add binary, infrastructure pipeline, data and identity to the mix, you end up with a vast array of tooling and a huge set of policies that every change needs to be assessed against.

Yes, you can scan containers on a periodic basis before deploying to production and that might be OK if you are deploying one release per month. However, even then you could argue that by the time the vulnerabilities are picked up, it’s too late and you have already lost the battle on efficiency.

The very reason you went with a microservice-based architecture is so that it’s easy to scale and quick to build and deploy. Ideally, you would want to be releasing features rapidly for your customers. This means a very high degree of change, which in turn means the point-in-time assessment you did will simply not be fit for purpose in terms of protecting your assets. Every commit has the potential of exposing a new risk, which may end up being discovered too late if left to point-in-time assessments.

Instead, if security is orchestrated seamlessly and asynchronously through tool-agnostic interfaces across your CI/CD pipelines, delivering the outcomes required to secure your digital assets from the first commit right through to the production deployments and beyond, you have truly adopted DevOps. In this scenario, every change is assessed in real-time, and its impact is projected across to development, operations, and security with a clear call to action.

This approach also provides a more extensible and scalable framework for organizations as they advance through their journey of DevSecOps adoption.

Summary

No development team deliberately sets out to build and deploy an insecure application. The reason applications with known vulnerabilities are deployed so often is because the cognitive load associated with discovering and remediating them is simply too high. The average developer can only allocate 10% to 20% of their time remediating vulnerabilities. The rest of their time is spent either writing new code or maintaining the application development environment used to write that code. If organizations want more secure applications, they need to find ways to make it easy for developers to correlate, prioritize and contextualize the vulnerabilities as they are being identified. Most of the time when developers are informed a vulnerability has been discovered in their code, they have long since lost context.

Vulnerabilities need to be immediately identified at the time code is written, builds are created, and pull requests are made – and identified in a way that is actionable. Otherwise, that vulnerability is likely to be thrown atop the massive pile of technical debt that developers hope they’ll one day have the time to address.

At this juncture, it’s only a matter of time before governments around the world enact legislation that will hold organizations more accountable for the security of the software they build and deploy. Savvy DevSecOps teams already recognize that existing approaches to managing DevSecOps workflows will need to be revamped to address these requirements. The challenge and the opportunity now is to put in place a scalable security orchestration framework that eliminates friction in DevSecOps workflows and drives efficiency through real time assessments, ensuring that your software is secure by default.

March 25, 2024
OpenTelemetry Project Maintainers Add Code Profiling Capabilities

This week at the KubeCon + CloudNativeCon Europe conference, the maintainers of the open source OpenTelemetry observability agent software project being advanced under the auspices of the Cloud Native Computing Foundation (CNCF) revealed they have added profiling capabilities to enable DevOps teams to identify the root cause of issues down to a specific line of code.

Austin Parker, director of open source for Honeycomb and a maintainer of OpenTelemetry, said while several providers of IT observability platforms have in the past year added code profiling capabilities, it makes more sense to provide this as a capability surfaced by OpenTelemetry in a way that any IT platform can easily consume.

The overall goal is to reduce the amount of time it takes to troubleshoot IT issues by identifying not just the code at issue but also the development team responsible for it, noted Parker.

OpenTelemetry defines a format that makes it simpler to centrally manage the collection metrics, logs and traces using open source software. In general, OpenTelemetry continues to gain traction as DevOps teams look to rationalize the amount of commercial agent software they need to deploy, update and manage.

In effect, OpenTelemetry is aggregating agent software that, in terms of observability, provides no differentiated value, said Parker. That’s critical because as application environments become more complex, it’s all but impossible to manage them without being able to broadly apply the agent software required to instrument them.

In the future, OpenTelemetry is expected to add additional capabilities such as data pre-processing, filtering or routing to redact sensitive data in a way that still fosters observability.

Ultimately, as OpenTelemetry becomes more widely employed, it makes it significantly easier for developers to understand how their application is performing, noted Parker. Over time, armed with those insights, developers will build and deliver higher-quality applications, he added.

It’s still early days as far as adoption of OpenTelemetry is concerned, but it’s clearly well on its way to becoming a de facto standard for enabling observability. The number of developers who have access to not just metrics and logs but also traces should steadily increase as the cost of instrumenting code declines.

Of course, finding a way to collect that data inexpensively will also be crucial for applying artificial intelligence (AI) to DevOps workflows. After all, before an AI model can be trained, it needs access to verified data.

In the meantime, DevOps teams might want to consider at what rate they can replace existing commercial agents deployed across the enterprise. Much of that agent software is more mature than OpenTelemetry, but over time, the economic arguments for using open source agent software become compelling. Many of the providers of observability platforms have already made support for OpenTelemerty available by default.

Regardless of approach, the need for observability has always been apparent. The issue is that there is a world of difference between monitoring a pre-defined set of metrics and being able to launch queries to identify the root cause of an issue that metrics alone are not going to be able to surface.

March 22, 2024
Time to Extend DevOps Visibility to the Network Edge

Performance issues regularly confound DevOps teams after an application is deployed, despite the number of tests run before deployment. Upon further investigation, the issue that is most frequently overlooked is the distributed nature of the application itself. End users accessing applications from multiple locations are never going to have the same level of internet service, so software that may be running perfectly well in locations that are major urban technology hubs, such as New York, San Francisco or London, is not necessarily going to provide the same experience in farther-flung regions, especially rural areas that are halfway around the world.

All internet services are not created equal. Intermittent network latency issues can have a profound impact on application performance simply because of the laws of physics. The farther away an end user is from the data center where the application is hosted, the more likely it becomes network latency will adversely affect application performance.

Unfortunately, most DevOps teams lack any visibility beyond the data center environment where the application is running. Naturally, the first step toward gaining that level of insight starts at the edge of the network.

Internet Stack Visibility

There is a wide range of services that make up an internet stack, each of which can adversely impact the performance of distributed applications. Everything from virtual private networks (VPNs) and content delivery networks (CDNs) to domain name system (DNS) servers and switches adds latency that, depending on how network traffic is being routed, can have a major impact on application performance. Any one of these services might be experiencing an outage or, more challenging to detect, intermittent degradations that unpredictably impact application performance. Whether it’s a software-as-a-service (SaaS) application being accessed in the cloud or an internet-of-things (IoT) platform running at the network edge, pinpointing which internet service might be the root cause of an issue requires the ability to closely monitor and observe that traffic.

Without any visibility into the internet stack, surfacing the insights needed in a way that provides the context required to optimize and troubleshoot a specific application becomes an impossible challenge. All the time and effort spent building an application can come to naught simply because no one understood the impact network latency would have on the application before it was deployed. Even after an application is deployed, any change to the way network traffic is being routed can have major consequences. Without any means of understanding what changes may have occurred, it might be weeks or months before an organization realizes it may need to log a service request or switch service providers.

Any truly comprehensive approach to observability clearly needs to include an analysis of the internet services upon which organizations crucially depend to ensure a consistent application experience is delivered and maintained. Depending on how the IT team is structured, those insights can be fed into an observability platform that the organization has adopted or correlated using an internet performance monitoring (IPM) platform.

DevOps Needs NetOps

Regardless of the approach, it’s never been more crucial for network operations (NetOps) and DevOps teams to be able to align their approaches to monitoring distributed computing environments. More latency-sensitive applications are being deployed than ever. In an ideal world, DevOps teams working closely with NetOps should be able to programmatically provision and alter networking services as needed. Not every internet service can be adjusted via an application programming interface (API), but most NetOps teams have access to consoles that enable them to upgrade to different service levels when needed. At the very least, a service provider will work with them to make any changes required.

Of course, it becomes more challenging to negotiate if the NetOps team doesn’t know which element of the Internet stack needs to be upgraded. A service provider that typically doesn’t have any visibility into the application might spend weeks trying to determine which element of the Internet stack they manage is at fault. IT teams that have their own means of monitoring a set of internet services are going to be able to work with their service providers to resolve those issues much faster.

Summary

Savvy IT teams are all too aware of how prone internet services are to outages and brownouts, so as a rule, they tend to make sure network traffic can be rerouted from one service to another. That approach also provides the added benefit of maintaining some pricing leverage as they negotiate fees for the services provided.

It’s also worth remembering providers of networking services that know a customer is locked in may not have as much incentive to make sure networking services are running at peak efficiency. The only way an IT team is ever going to know what level of service is being delivered at any given time is, of course, to generate an IPM report that precisely details what services were actually available at what level of performance. Otherwise, they are wholly dependent on the report a service provider presents alongside their monthly bill.

Just as critically, the reports generated by an IPM platform also surface insights that, when shown to application developers, create a potential opportunity to optimize their code. Without that level of analysis, every developer will assume the network services are either broken or poorly managed. The challenge and the opportunity now is to provide internet performance intelligence in a way that every stakeholder cannot just act on but, just as critically, implicitly trust.

March 22, 2024
Edge Computing Requires DevOps at Scale

The rise of edge computing is about to drive a long-overdue convergence of DevOps, data engineering, security, networking, operational technology (OT) and machine learning operations (MLOps) best practices, which will ultimately make IT teams more responsive than ever to the needs of the business.

Historically, each of these IT disciplines has tended to work in isolation, with deliberate handoffs being made between the various teams employed to provision and maintain various services. The challenge is that as more applications are deployed at the network edge, the need for IT teams to work hand-in-glove is crucial. Applications not only need to be deployed across a much more distributed computing environment, but they also need to be regularly updated and patched.

At the same time, increasing amounts of data are being processed and analyzed at the point where it is collected and consumed. More stateful applications running at the network edge are passing aggregated analytics data back to applications running in the cloud or on-premises IT environment. That shift requires a fundamentally different approach to how storage is managed across multiple edge computing environments and the various federated backend systems spanning the enterprise.

Organizations will typically need to store files locally on edge computing platforms while also storing copies of those files in the cloud using an S3-compatible storage service that provides access to inexpensive object-based storage to enable everything from data protection to driving artificial intelligence (AI) applications. That multi-protocol capability creates an opportunity to not just build new stateful distributed applications but also modernize existing applications regardless of what protocols are being used to store data.

Most edge computing platforms today are managed by OT teams that report directly to the leaders of specific business units. However, as more of these platforms are connected to networks, the greater the imperative to centrally manage them as part of any effort to lower the total cost of IT. After all, the most expensive element of IT remains the cost of the labor still required to manage it.

Naturally, sharing data in near real-time across a distributed computing environment puts more pressure on network operations (NetOps) teams. Instead of updating applications in batch mode once a day, data continuously streaming from thousands of edge computing platforms needs to be synchronized with multiple applications to surface the latest, most accurate insights.

More challenging still is that much of that data is being used to train AI models that ultimately require the associated inference engine created to be deployed at the edge. To process terabytes of data, the underlying platform required to support those inference engines also will need to be measured in terabytes. Over time, AI models tend to drift or, in the case of generative AI, outright hallucinate, over time. Data scientists will need to work closely with DevOps teams to replace those AI models because, unlike a traditional application, they can’t be updated via a patch. The entire AI model needs to be replaced with one that is not just more reliable but also—and just as importantly—safer.

Finally, the need to secure edge computing platform is nothing short of critical. Cybercriminals tend to view these platforms as gateways to the rest of the enterprise. Each new edge computing platform deployed extends the size of the attack surface that needs to be defended. Unless cybersecurity teams are deeply involved in building and deploying edge computing applications and platforms, it’s only a question of when, rather than if, a breach will occur.

It may be a while before these various IT fiefdoms converge, but at this point, it is all but inevitable. In fact, it will soon be difficult to distinguish between edge computing and the rest of the IT environment as event-driven applications spanning multiple platforms become more the norm than the exception.

DevOps teams, as part of their relentless commitment to automation, should naturally be at the forefront of these efforts. Rather than confining DevOps principles to how applications are built and deployed, the time has clearly come to automate IT workflows on an end-to-end basis. The only way to programmatically manage edge computing at scale is to also apply DevOps principles at scale. Also known as platform engineering, that approach enables organizations to manage distributed IT environments at scale without requiring organizations to hire a small army of IT teams to manage it all. There will always be a need for specialized expertise, but as IT continues to evolve the siloes that slow down the pace of innovation will need to come down.

Not every member of an IT organization needs to know how to expertly manage every task, but they should be familiar with the inherent dependencies that exist between applications, networking, storage and cybersecurity to advance the goals of businesses that have never been more dependent on IT.

Each organization needs to determine the pace of convergence that best suits its needs, but the faster it occurs, the greater the return on the investment in edge computing will become. The challenge and the opportunity, as always, will be managing yet another major IT transition, where the issues that need to be resolved are as much cultural as they are technical.

March 22, 2024
How Generative AI Enables Unified Continuous Testing Platforms

It is time for a unified continuous testing platform.

Existing software testing platforms and tools specialize in testing activities for a subset of the different stages of value streams. The landscape of software testing tools includes a wide range of solutions, each with its strengths, focusing on specific aspects of software development, delivery and operations.

For instance:
• Unit Testing Tools focus on the earliest stages of development, allowing developers to test individual units of code for correctness.
• Integration Testing Tools aim to test the interactions between different modules or services within an application.
• System Testing Tools are designed for end-to-end testing of the complete system before it goes live.
• Performance Testing Tools assess the application’s behavior under load and stress conditions.
• Security Testing Tools focus on identifying vulnerabilities within the application.
• User Acceptance Testing (UAT) Tools facilitate the final testing phase, where end-users validate the solution against their requirements.

The challenge with unifying these diverse testing needs into a single continuous testing platform is multifaceted. Such a platform needs to seamlessly integrate with a wide variety of development tools and environments, support different testing methodologies, and be flexible enough to adapt to different organizational processes and quality standards.

While there are Continuous Integration/Continuous Deployment (CI/CD) tools that incorporate multiple testing stages, they often do so by integrating with specialized testing tools rather than offering a unified testing solution themselves. These CI/CD tools are closer to orchestrating the various testing activities rather than unifying them under a single platform’s capabilities.

However, the concept of a fully unified continuous testing platform, covering every stage from requirements through to deployment and in-production testing, represents a significant opportunity for innovation in the software development tools space. Achieving this would likely require leveraging advancements in areas like generative AI, as discussed in my prior blog Applying AI/ML to Continuous Testing to create adaptable, intelligent testing processes that can cover the full spectrum of testing needs in a cohesive manner.

Benefits of a Unified Continuous Testing Platform

Improved Efficiency and Speed: A unified CT platform integrates testing across all stages of the software development lifecycle (SDLC), significantly streamlining the testing process. This integration facilitates automated, continuous feedback loops that promptly identify and address defects, allowing development teams to quickly iterate and improve. As a result, teams can release new features and fixes more rapidly, accelerating the product’s time to market and enhancing responsiveness to customer needs and market changes.

Enhanced Quality and Reliability: By ensuring comprehensive and consistent testing across the entire application, a unified CT platform plays a crucial role in identifying and mitigating issues early in the development cycle. This early detection helps maintain a high level of software quality and reliability, leading to increased user satisfaction and trust in the product. The consistent application of quality standards across all testing phases contributes to a robust and dependable software product.

Cost Savings and Reduced Maintenance: Automating the testing process through a unified CT platform not only optimizes resource utilization but also significantly reduces the costs associated with manual testing and late-stage defect remediation. Early defect detection translates to lower fix costs, and the streamlined process decreases the overall time-to-market. Additionally, high-quality, reliable software requires less maintenance, further reducing long-term costs and freeing up resources for innovation and development efforts.

Improved Collaboration Across Teams: A unified CT platform fosters a culture of collaboration and transparency among development, testing, and operations teams. By providing a common framework and tools for all testing activities, it breaks down silos and enables seamless communication and collaboration across the SDLC. This improved collaboration ensures that teams are aligned with the project goals, can share insights more effectively, and work together to identify and solve problems more efficiently, leading to better outcomes.

Together, these benefits highlight how a unified Continuous Testing platform can transform the software development process, making it more efficient, cost-effective, and collaborative while ensuring the delivery of high-quality and reliable software products.

While many tools address specific parts of the testing lifecycle, the vision of a single platform that seamlessly unifies all stages of testing from requirements to in-production remains an ambitious goal. It’s a gap in the current technology landscape that presents both a challenge and an opportunity for future development.

Challenges for Unified Continuous Testing Platforms

Creating a continuous testing platform that unifies test activities for all stages of the end-to-end value stream is a complex challenge due to several factors:
1. Diversity of Technologies and Tools: Modern software development environments are highly diverse, incorporating various programming languages, frameworks, and technologies. Creating a platform that seamlessly integrates with all these technologies is challenging.
2. Complex Integration Points: Continuous testing needs to integrate with multiple stages of the development pipeline, including development, deployment, and operations. Each of these stages might use different tools and processes, making it hard to create a one-size-fits-all solution.
3. Varying Quality Metrics: Different teams and projects may have different definitions of quality, success criteria, and performance metrics. A unified testing platform needs to be highly customizable to cater to these varying needs.
4. Change Management: Adopting a new platform requires changes in the organization’s processes and workflows. Resistance to change is common in organizations, and transitioning to a new way of testing can be met with skepticism and inertia.
5. Scalability and Performance: Ensuring the platform can scale to handle the testing needs of large organizations with thousands of tests running concurrently is a technical challenge. Performance issues can become a bottleneck, affecting the overall efficiency of the development process.
6. Security and Compliance: Integrating testing across all stages of development also introduces security and compliance challenges. The platform must ensure that sensitive data is protected and that testing practices comply with regulatory requirements.
7. Cost and Resource Constraints: Developing, maintaining, and supporting a unified continuous testing platform requires significant investment. Organizations might be hesitant to commit the necessary resources without clear evidence of return on investment.
8. Evolution of Practices: Software development practices and tools are constantly evolving. Keeping the platform up-to-date with the latest practices and technologies requires ongoing effort and innovation.

Despite these challenges, there is a growing recognition of the value of continuous testing throughout the software development lifecycle. Some companies and open-source communities are making strides in this direction, creating more integrated and flexible testing solutions. However, achieving a fully unified platform that addresses all these challenges is an ongoing effort and represents a significant opportunity for innovation in the software development and testing industry.

Generative AI Can Facilitate a Unified Continuous Testing Platform

Generative AI can play a significant role in overcoming the challenges associated with creating a continuous testing platform that unifies test activities for all stages of the end-to-end value stream. Here’s how generative AI might address each of the challenges:
1. Diversity of Technologies and Tools: Generative AI can be trained on a wide range of programming languages, frameworks, and technologies to understand and generate code or testing scripts. This capability allows it to adapt to different environments and create testing materials that are compatible with various tools and technologies.
2. Complex Integration Points: AI can analyze the workflow of development pipelines and suggest optimal integration points for testing. By learning from different CI/CD (Continuous Integration/Continuous Deployment) configurations, AI can recommend best practices for integrating testing seamlessly into existing workflows.
3. Varying Quality Metrics: Generative AI models can be customized to understand and apply different quality metrics and success criteria based on project-specific requirements. By training on diverse datasets, these models can adapt to various definitions of quality and generate relevant tests or analysis.
4. Change Management: AI can assist in the change management process by simulating the outcomes of adopting new testing platforms, thereby providing evidence-based benefits and mitigating resistance to change. Furthermore, AI-driven analytics can highlight the efficiency gains and quality improvements to support the transition.
5. Scalability and Performance: Generative AI can optimize testing processes by identifying redundancies and suggesting improvements, thus enhancing performance. Additionally, AI can dynamically allocate resources based on testing needs, ensuring scalability without compromising efficiency.
6. Security and Compliance: AI models can be trained to identify and flag potential security and compliance issues in the testing process. By continuously learning from the latest security standards and compliance regulations, AI can help ensure that testing practices meet the necessary requirements.
7. Cost and Resource Constraints: By automating the generation and optimization of test cases, generative AI can significantly reduce the manual effort required, lowering costs and resource demands. AI can also help prioritize testing efforts based on risk assessment, ensuring that resources are focused where they are most needed.
8. Evolution of Practices: Generative AI models are inherently adaptable and can continuously learn from new development practices, tools, and technologies. This ensures that the testing platform remains up-to-date with the latest advancements in software development.

Generative AI has the potential to transform continuous testing by providing adaptive, efficient, and intelligent solutions to the complex challenges of unifying test activities across the end-to-end value stream. However, realizing this potential requires careful design, extensive training of AI models, and ongoing management to ensure that the AI systems remain effective and aligned with evolving testing needs.

Summary: Call to Action

The need for a unified Continuous Testing (CT) Platform has never been more urgent. Current testing platforms, each adept in their niche, cover only fragments of the software development lifecycle (SDLC), leading to a disjointed and inefficient testing process. This fragmentation not only slows down development and deliveries but also compromises the quality and security of the final product. The dream of a single platform that seamlessly integrates all stages of testing, from requirements through to deployment and in-production testing, represents a monumental leap toward efficiency, security, and quality in software development.

The challenges in creating such a platform are manifold, ranging from the diversity of technologies and tools to the evolving nature of software development practices. Each challenge, from integrating diverse technologies and managing change within organizations to ensuring scalability and compliance, adds complexity to the development of a unified CT platform. Yet, the potential benefits of overcoming these hurdles are immense, promising a significant boost in the speed and quality of software delivery. The industry’s recognition of these benefits is growing, evidenced by the efforts of some companies and open-source communities moving towards more integrated and flexible testing solutions.

Generative AI emerges as a beacon of hope in this quest, offering innovative solutions to the multifaceted challenges of unifying test activities. By harnessing the power of generative AI, the industry can address the diversity of tools, integrate complex testing stages, adapt to varying quality metrics, manage organizational change, scale efficiently, ensure security and compliance, and evolve with the practices of software development. The path forward requires a concerted effort, but investments in AI-driven testing innovations can realize the vision of a comprehensive, unified, continuous testing platform. This is not just an opportunity for enhancement but a call to action for the industry to redefine the future of engineering platforms.

March 21, 2024
DevOps: Addressing Communications Failures

The more complex software engineering becomes, the more apparent it is becoming that the communications processes relied on to manage DevOps workflows are fundamentally flawed. Every DevOps team has experienced multiple instances where, for example, a software package needed to build or deploy an application isn’t where it’s supposed to be when required. The outcome is additional delays that, in hindsight, were completely avoidable.

Fortunately, organizations are starting to realize that the tools used to build and deploy software create metadata that can be tracked and also harnessed to optimize workflows. For example, TestifySec’s Witness, an open source tool to generate and verify attestations that provide a verifiable record of the steps taken to build software, including the materials used and commands run during a DevOps workflow. A companion open source Archivista project manages storage, retrieval and retention of software build pipeline attestations and trusted telemetry observed by Witness.

“It’s really about being able to prove the provenance of your artifacts,” said TestifySec CEO Cole Kennedy. “It’s about communication.”

These and other tools are at the forefront of a broader effort to harness the metadata software build tools already generate in a way that not only helps ensure tasks have been completed but, ultimately, serves to make the software supply chain more secure.

“To effectively improve software supply chain security, it’s important to be intentional about observing DevOps workflows rather than hope to reconstruct data from workflows after the fact,” said Mitch Ashley, principal analyst for Techstrong Research.

In fact, as regulations requiring organizations to manage software supply chains continue to become more stringent, it’s now only a matter of time before compliance teams require software engineering teams to be able to document every interaction that occurred across the entire software development life cycle (SDLC).

The DevOps Documentation Challenge

DevOps teams today routinely employ multiple tools to create software artifacts, manage software engineering pipelines and, ultimately, deploy software. The level of DevOps maturity will vary widely from one organization to another, but they all have one thing in common: Somewhere, a project manager is trying to keep track of the interactions between those tools to ensure application delivery schedules are met. The challenge is, historically, those project managers have been forced to rely on manual data entry to capture those interactions in a way that can be used to verify a task was completed. In an ideal world, the tools themselves would automatically share the data required with an application that makes it easy to determine which tasks have actually been completed.

In the absence of that level of automation, it should come as no surprise that projects are delayed simply because one critical task or another was not completed in time for the next phase of the software development process was unable to begin on time. Inevitably, for want a proverbial nail, the battle is lost as delays that could have been avoided have a cascading impact throughout the software development life cycle.

Adding further insult to injury, no one effectively learned from that experience because the documentation that describes what occurred is either sparse or simply nonexistent. The truth is, software engineering teams are often at a loss when it comes time to explain to business and IT leaders what went wrong because there is no way to determine beyond a shadow of a doubt what occurred and precisely when during the software development process.

The Compliance Challenge

This lack of transparency into software development workflows has been an open secret for longer than most IT leaders care to admit. However, as software supply chain security concerns continue to mount, more organizations are being required to document the software engineering workflows that were relied on to build their applications. In the event of an audit, any inability to provide that documentation will only increase the size of the fines that might be levied.

The more complex the applications being deployed become, the more probable it becomes a critical event upon which an audit revolves will not have been captured by manual processes that, by definition, are prone to human error.

Unfortunately, as most IT leaders already know well, when it comes to compliance, there’s not a lot of forgiveness for human frailty.

Automation

As is almost always the case when it comes to DevOps workflows, the best path forward is to almost always rely on increased automation. One of the primary reasons most of these workflows are not well documented is that humans, in general, and application developers, in particular, are not overly fond of data entry. No one wants to spend time keying in data they know already resides in another application. In addition to wasting time, there’s also a high probability mistakes will be made as data is manually copied from one application to another.

Validating events that occur is how organizations manage risk. Otherwise, business and IT leaders are hoping that all the tasks that need to be completed to enable an application to be delivered on time are actually done. Hope is not a substitute for a platform that automatically captures, analyzes and validates each step that occurred to determine actual intent versus simply presenting a series of logs that don’t provide nearly enough meaningful context.

Summary

Slowly but surely, the processes organizations rely on to manage software are becoming more advanced. Capabilities that a few short years ago would have seemed far too difficult to provide are now readily accessible. Capturing metadata in a way that enables organizations to automate workflows at unprecedented scale is increasingly becoming commonplace.

The challenge and opportunity now is finding the simplest means possible to harness it at a time when application environments are becoming too complex to manage otherwise.

March 19, 2024
JFrog Survey Surfaces Raft of DevSecOps Challenges

A global survey of 1,224 security, development, and IT operations professionals published today by JFrog finds that 60% typically spend four days or more remediating application vulnerabilities in a given month.

JFrog security researchers, however, noted it’s probable most of those vulnerabilities are not as severe as they have been rated. After analyzing 212 vulnerabilities, the JFrog Security Research team downgraded the severity of 85% of the vulnerabilities rated as critical and 73% of one rated as high.

JFrog researchers also found that 74% of the reported common vulnerabilities and exposure (CVEs) with high and critical CVSS scores assigned to the top 100 Docker Hub community images weren’t actually exploitable.

Only 17% of the vulnerabilities analyzed enabled remote code execution, compared to 44% that could enable a denial of service (DoS) attack, the researchers noted.

Stephen Chin, senior director for developer relations for JFrog, said that analysis suggests organizations are arguably spending too much time on application security at the expense of productivity. Tools that generate alerts based on severity rather than also rating exploitability are creating too much noise simply because they don’t provide enough contextual analysis, he added.

Conversely, in the absence of that level of detailed analysis, it’s possible a low-level vulnerability that might be especially severe in a specific IT environment, also known as a false negative, might not be further investigated, noted Chin.

Security concerns, however, also limit innovation, with 40% of the survey respondents noting that because of security reviews, it typically takes a week or longer to get approval to use a new package/library.

Nearly half of IT professionals (47%) say they use between four and nine application security tools, with 90% reporting they use some type of tools that rely on machine learning algorithms or other forms of artificial intelligence (AI) to scan, identify and remediate vulnerabilities.

Less than a third (32%), however, work for organizations that use AI to write code. That relatively low level suggests many organizations are still not comfortable with the quality of the code generated, noted Chin. Of course, individual developers may still be using those tools without approval.

More than half (53%) say their organization uses between four to nine programming languages, with nearly a third (31%) reporting they use 10 or more.

The most frequently used tools are static application security testing (61%), dynamic application security testing (58%), software composition analysis testing (56%), and application programming interface (API) security (56%).

A full 89% said their organization has adopted a security framework such as OpenSSF or Supply-chain Levels for Software Artifacts (SLSA).

Finally, the survey finds little consensus concerning when to run security scans, with 42% saying it’s best to perform security scans as code is being written versus 41% that prefer to perform scans on new software packages before installing them. A total of 41% said runtime is the least desirable place to run scans. More than half (56%) said their organization applies security scans at both the code and binary scanning levels.

The degree to which DevSecOps best practices are being adopted will naturally vary from one organization to the next. The challenge is to prioritize remediation efforts in a way that enables developers to spend more time writing, for example, business logic versus patching vulnerabilities are not actually a material threat. In that context, the quality of the vulnerability analysis generated by scanning tools matters because of the unnecessary toil that might be eliminated, noted Chin.

It may be a while before most organizations are able to strike the right balance when it comes to DevSecOps, but as a greater appreciation for the nuances of application security continues to take hold, the less stress there should hopefully one day be for all concerned.

March 19, 2024
Five Great DevOps Job Opportunities

devopsy.kinsta.cloud is now providing a weekly DevOps jobs report through which opportunities for DevOps professionals will be highlighted to better serve our audience.

Our goal in these challenging economic times is to make it easier for DevOps professionals to advance their careers.

Of course, the pool of available DevOps talent is still relatively constrained, so when one DevOps professional takes on a new role it tends to create an opportunity for others.

The five job postings shared this week are selected based on the company looking to hire, the vertical industry segment and, naturally, the pay scale being offered.

We’re also committed to providing additional insights into the state of the DevOps job market. In the meantime, for your consideration:

LinkedIn

Zscaler
Remote
Staff DevOps Engineer
$165,750 to $195,000

Indeed.com

National Hockey League (NHL)
New York, New York
DevOps Engineer
$140,000 to $175,000

CareerBuilder.com

World Wide Technology
Chantilly, Virginia
DevOps Engineer
$120,000 to $220,000

SimplyHired.com

Hard Rock Digital
Austin, Texas
Senior DevOps Engineer
$116,000 to $147,000

Dice.com

HCL America
Seattle, Washington
Senior DevOps Architect
$115,000

March 18, 2024