That’s the question Goldman Sachs asked in an influential report this summer.
It’s safe to say we’ve entered a trough of AI disillusionment. Some companies are laying off staff, while others are being targeted for acquihire (for example Character AI by Google, Inflection by Microsoft, and Adept by Amazon). These factors combined with swings in tech stocks indicate that the market has yet to make up its mind when it comes to valuing AI.
Despite the uncertainty, genAI has gained traction in several sectors, and engineers in particular have embraced AI coding tools like GitHub Copilot. Stack Overflow’s 2024 Developer Survey found that “76% of all respondents are using or are planning to use AI tools in their development process this year.”
With AI coding tools, software companies took the approach of testing first and asking questions later. Those companies are now taking a microscope to their investments and need to justify whether their spending on coding assistants is paying off.
Continued Uncertainty Around Performance and Pricing
Earlier this summer, Jellyfish introduced the Copilot Dashboard to measure the impact of the most widely adopted genAI coding tool. We’ve since gathered data from over 4,000 developers at more than 100 companies, giving us a representative sample of how engineering organizations are using Copilot and what impact it’s having on production. No two teams are the same, but the aggregate data can help engineering and business leaders understand whether they’re getting adequate return on their AI investments.
Without measurement, engineering teams can only gauge the value of their investment in terms of engagement. Are the engineers using Copilot? Sure. But are they faster or more productive? Is the code better, or is that speed coming at the expense of quality?
Pricing for these tools is another potential variable. In October 2023, The Wall Street Journal reported that Microsoft was losing $20 per month per user on its GitHub Copilot product. GenAI coding assistants like Copilot rely on LLMs to drive their service, and the cost of querying those LLMs isn’t cheap! Providers of AI coding tools will inevitably make corrections to their pricing, and software companies will have to decide all over again whether they’re getting value for their investment.
After analyzing aggregate data from Jellyfish’s GitHub Copilot Dashboard, here’s what we found:
Coding is faster, leading to time savings: Across the board, developers are writing code 23% faster. This impact varies by role. For example, mobile developers and backend developers experience more substantial gains, with coding speeds improving up to 42% and 38%, respectively.
Coding throughput is increasing: Engineers using Copilot saw a 20% increase in pull requests (PRs) created across all engineering work, meaning, in layman’s terms, that these engineers are pushing more code (and therefore more product) out into the world.
Junior developers see the largest increase in speed: Across all engineers, Copilot is giving junior developers a 4X boost in coding speed. This implies that Copilot is more useful for automating entry-level tasks and less useful for the more complicated work tackled by senior developers.
Where We Go From Here
We’re in the trough of disillusionment of the genAI hype cycle but there is a bright spot. The technology will continue to develop, and more targeted solutions like AI agents have the potential to disrupt markets more than general AI tools.
Copilot is a transformative tool for software engineering, but organizations must be realistic about its limitations. Coding is only a fraction of what engineers spend their time on in the entire software development life cycle. The first measurements we’re seeing of Copilot adoption show meaningful boosts across different roles and workflows in software organizations. Taken together, these boosts add up to time and cost savings that more than justify the investment now while setting organizations up to realize larger downstream gains in the future.
Regardless of what happens in the months to come, companies can’t afford to experiment without measurement. The “trial and error” phase for genAI is over; we need to verify whether the investments are paying off. Experiment, measure, adjust, repeat.
As more organizations focus on digital transformation as the centerpiece of their growth strategy, software developers and IT teams are under heightened pressure to perform—quickly and efficiently.
That begs two questions: Can we do better? And if so, how can we do better?
The answers can be elusive, especially for DevOps teams that have over-relied on processes and models that, while previously successful, may not be as relevant for today’s projects. Of course, the point about relevance has been further magnified during the pandemic, when sudden shifts to remote working left many teams scrambling to ensure they had the capacity and infrastructure to support massive work-from-home cohorts.
Software development and IT operations support teams that are winning by doing better in this new era are truly embracing Agile. And that means taking to heart the definition of “Agile,” which is a mindset of values and principlesto continually evolve your way of working (WoW) to ensure your productivity and value are tailored to fit the tasks at hand.
Conversely, teams that have relied on their go-to framework or approach for every assignment are often missing the boat with respect to efficiencies, effectiveness and even ROI. This is because they aren’t assessing whether their proven WoW is useful for their current projects.
That can lead to underwhelming results. A report from Digital.ai revealed that while over 90% of software development teams take an Agile approach to their work, a whopping 96% of their digital transformation projects have failed or been delayed. Clearly, that suggests Agile has worked for them in the past,but the nature of digital transformation projects is calling for teams to adapt to new or hybrid approaches that may differ from their go-to WoW.
Challenging? Yes. Doable? Also Yes
While there is no quick fix to attain true flexibility amid complexity, the journey begins with the recognition that, yes, you can do better. And to do better—to become an adaptive, lean, responsive and learning organization under any circumstances—there are a number of steps to take to ensure your journey to Agile freedom is successful.
Here are five that I believe must be considered for DevOps teamsto become better and more effective:
Consider all options. Agile has been practiced for more than two decades, and literally hundreds of frameworks and approacheshave been developed and implemented. If you examineyour WoW in a context-driven, pragmatic manner you are more likely to select the best-fitting technique given the situation.
Do your homework.Your team leaders may not be familiar with the breadth of the Agile, traditional and hybrid frameworks at their disposal. But there’s help available. Seek out online courses and certifications designed to help organizations and individuals explore and navigate the multitude of Agile frameworks that they may currently be overlooking.
Focus on value. It’s great if your gut tells you that your Agile framework is reaping real results, but can you prove it? If your senior leadership has invested in Agile training, coaching and certifications, they’ll demand to see some tangible returns. With well-chosen Agile approaches that match the work required, you will be able to deliver the proof that Agile is driving measurable value that matters. Even further, learning which metrics to use in different situations can help measure this value.
Sidestep bottlenecks. By choosing the best WoW for the project at hand, you are more likely to eliminate the wasteful or inefficient processes that can add hours or days to any assignment. Understanding how to unclog these pipes is critical in improving the flow of value to your customers.
Motivate with morale. If your team is frustrated and morale is low, productivity, the quality of the work and your retention rate will be, too. But, if you’re able to do Agile well, this can help your teams be happier, more productive and more successful in accomplishing project goals. Ultimately, that means a more joyful customer base, too.
Conclusion
In the long run, the framework or approach you choose isn’t as important as your willingness to be open-minded in applying Agile frameworks to projects on a case-by-case basis. As you become more comfortable with new or different approaches, you will promote continuous improvement through carefully considered experimentation.
That means you’ll become a learning organization, one that is open to change and evolution. With that mindset, your development and IT ops team will do better—guaranteed. Because true agility comes from freedom, not frameworks.
When it comes to accelerating customer value, it’s dangerous to focus only on the development and delivery of a digital product. For many enterprises, most time is invested before a single line of code has been written. That explains the recent focus on value stream management (VSM)—it’s forcing us to think beyond Agile and DevOps and consider the whole system, not just one stage in a complex process.
By extending our visibility further upstream where work originates, we can begin measuring the value of creating and protect workflows end-to-end to meet business outcomes sooner. Thinking about how software delivery work moves between all teams in the value stream from request to operation leads us to the significant realization that our tooling pipeline is our most important product and we should treat it as such.
The Rise of Value Stream Management
The growth of value stream management in software delivery has emerged because Agile and DevOps are not enough to remain competitive against the breakneck speeds of the tech oligopoly. The progressively maturing practice takes a systematic approach to continuously measuring and improving end-to-end flow to accelerate time-to-value. Consequently, new tooling is emerging as we think end-to-end in how we collaborate and work faster together—and measure our productivity and effectiveness.
Measuring and understanding end-to-end flow isn’t easy. Software delivery is highly technical and creative knowledge work that can span thousands (or tens of thousands) of IT practitioners. Information is scattered across heterogenous toolchains comprising discipline-specific tools. In fact, some 42% of large-scale software delivery organizations use four to 10 core collaboration tools for portfolio management, design, enterprise agile planning, testing and ITSM, and dozens of additional tools to support the CI/CD pipeline and operations.
Mik Kersten, “Mining the Ground Truth of Enterprise Toolchains, IEEE (Volume 35, Issue 3, May/June 2018, p12-17
Measuring flow, therefore, involves tracing constituent units of value creation and protection as they traverse all team tooling in the form of artifacts such as activities, capabilities, epics, features, stories, bugs and incidents. Yet for too long we’ve been too focused on “the right side” of the value stream, thinking about how products are built and delivered through Agile and DevOps tools, without visibility on how they’re requested, analyzed, processed and designed. Our ability to support customers is hindered because we don’t know where and why work is slowing down outside of development. Our product pipeline is broken, our visibility fractured.
A ‘Broken’ Product Pipeline
Traditionally, CI/CD tools include source code management, build, deploy, quality management and service management. Moving more to the right, you have release management and service management. But too often organizations have ignored integrating tools further to the left such as planning and portfolio management, product management, requirements management and risk management (static and dynamic security scanning tools). This impacts our ability to see and measure end-to-end flow.
In”Accelerate,” lead time is defined as the portion of the value stream from (last) code commit until deployment. While this is essential to improve, it only covers the time from the build being produced through deployment. We need to account for the time that elapses before it hits a developer’s backlog. To shift IT from a project to a product focus, you need to reduce end-to-end flow time by measuring earlier in the delivery life cycle by “shifting left.” As such, you need tooling across the whole value stream through the ideate, create, release and operate phases. A focus only on optimizing the CI/CD tooling and practices is too narrow a view that produces diminishing returns over time.
The Great Disconnect Between IT and the Business
Because of this lack of a holistic view, IT teams don’t feel like they are able to influence the business side of the house, where most of the work in progress (WIP) starts, such as requests for features or application changes. These intake processes many times result in teams taking on too much WIP and neglecting work. Many teams need to “stop starting” and to “start finishing” to deliver business value more quickly. Crucially, this is the area where business leaders have their own processes and where tools are less likely to be used; it’s a world dominated by spreadsheets, documents and PowerPoint presentations. This is where work is least visible and measurable. And this is precisely where we need VSM: to turn the light on starting with the customer needs so we can continuously improve end-to-end flow. This requires a culture shift from both IT and business/product owners. And that requires taking a broader, more holistic view of the tools and practices in the business space.
Architecting for Flow
An end-to-end integrated pipeline product can be architected for flow. For example, tools used for visual system management (e.g. Jira, Azure DevOps) are becoming more of a commodity with the real differentiation becoming how they can be integrated with tools used for portfolio, defect and service management, since those are the sources of work that will flow into product-based teams for development of business value and unplanned work. Teams should think about how they can work more closely with business portfolio teams on continuous flow (and funding) because eventually this work will become the next bottleneck that needs to be addressed on the journey to accelerate business delivery capability.
Keep an Eye on Value Protection Work
With end-to-end visibility into work across the pipeline, you can abstract that connected technical data into overarching value stream metrics—known as flow metrics—to identify areas where to run experiments to improve end-to-end flow. Here we mustn’t just focus on work that creates value but protects it, too.
We should provide our software delivery teams with the psychological safety to devote time to address debt and neglected work. Debt creates a drag on feature velocity because changes become more complex and take longer to make. In addition, with higher technical debt, more time is spent on addressing defects, which starves the capacity to perform feature work. As debt builds, the following is observed:
Flow velocity decreases.
Flow time increases.
Quality decreases as the code base becomes more fragile/costs increase.
All of these result in reduced happiness for both the customer and the team, increasing the risk of team burnout. There are many cautionary tales of this occurring where market leaders ended up losing considerable market share due to inattention to technical debt. These stories exemplify why it is necessary to observe flow distribution to ensure that prioritization decisions made with the product owners, include providing necessary capacity to address technical debt.
Applying the concepts of value stream architecture to your end-to-end pipeline is the key differentiator for organizations that are trying to make it through disruptive periods such as today, as COVID-19 widens the gap between those who have mastered software at scale and are thriving and the many enterprises struggling to pivot and adapt in the digital-first climate.
Many business leaders are planning to make DevOps the cornerstone of their IT transformation strategy. But, as surprising as it might sound, skepticism abounds regarding how major vendors address the DevOps transformation challenges—they confuse DevOps with continuous delivery (CD) infrastructure.
“We wanted DevOps and we got after a nine-month-long implementation a continuous delivery platform which speeds up IT operations without significant business value. How differently can you help?” the senior business executive of a major financial institution in New York sarcastically asked me during a conversation three months ago.
The business executive is right: DevOps isn’t a technology as I’ve been implementing it, it’s primarily an accelerated application delivery approach implemented in the form of a technology-enabled application delivery operational model. Adam Jacob, CTO of Chef, substantiates that fact in “The Secret of DevOps: It’s Always Been About People, not Technology,” when he says, “DevOps is about bringing together all the people you need to build and run your business effectively, and empowering them to move as quickly as possible towards their goals.”
Agile Admin’s Ernest Mueller in, “DevOps: It’s not Chef and Puppet,” confirms Adam’s point: “Chef and Puppet are individual tools that you can use to implement specific parts of an overall DevOps strategy if you want to use them—but that’s it. They are fine tools, but do not solve DevOps for you.”
How can we implement DevOps in line with business leader expectations? Three Amazon Web Services (AWS) Design Patterns I developed can help to get the best of DevOps on AWS cloud: Software Organization Factory, Digital Organization Factory and CD Infrastructure Factory. This article discusses the Software Organization Factory.
The Foundation of DevOps Value: Consequences to IT Service Delivery
Before delving into these AWS design patterns, a quick reminder of what DevOps is, its value propositions and, more importantly, what changes are needed in how IT services are delivered.
The Ignored DevOps Business Logic We All Must Consider
The combined effect of cloud adoption and the rising digital economy is changing competitive environments; it forces business leaders to focus on innovation and market responsiveness as the competitive differentiators.
The fact of the matter is, by facilitating access to technology, cloud accelerates entrance of several thousands of tech startups that take advantage of the recent technology innovations to substitute traditional products and services into digital. This creates a context where new entrants increasingly compete with established brands, forcing business leaders to seek value paradigms other than IT efficiency.
The business executive argues, “In these new markets, our concern is to increase customer value by delivering the right services to the right segments at the right moment. You guys need to change the IT paradigm; IT alone does not create that kind of value.”
Then, what should be changed in today’s IT approach?
The Two Dimensions of Modern IT: The Operational Model and the Continuous Delivery Infrastructure
The new IT paradigm advocated in my recent book, “Rethink Your IT with DevOps: The Complete Guide to Aligning Your IT to DevOps,” is the idea that business value results from two dimensions: organizational agility and technology efficiency. In other words, it defines modern IT as a two-dimension asset including the operational model and the continuous delivery infrastructure.
The timely delivery of innovation transcends IT issues; it not only cuts across your sales, marketing, post sales, application development and IT operations, but also demands concrete answers as to how to accelerate business prioritization, decision-making and problem solving. The operational model is where business and IT people, processes, practices and tools are combined to create agile and cross-functional environments. The following exhibit shows a typical DevOps operational model:
Typical DevOps Operational Model
Organizational flexibility for the sake of flexibility doesn’t make sense unless technology is leveraged to speed processes. This is where CD infrastructures come into play; throughout the delivery process, they build on IDEs to accelerate coding and testing, on versioning tools to convert software version into releases, on continuous integration (CI) tools to build releases and on delivery pipelines to ensure continuous delivery.
So, what is DevOps?
DevOps is a Technology-Enabled Application Delivery Model
DevOps is a combination of value, principles, processes and practices shared not only by developers and IT operations but also by the business. The goal is to help the organization deliver the right service to the right market segments at the right moment.
DevOps from the Business Perspective
As I’ve been implementing it, it’s a cross-functional application delivery model involving the business and IT and enabled by continuous delivery infrastructures.
By enabling and automating agile practices, the continuous delivery infrastructure makes the operational model agile and provides the business with the flexibility and velocity needed to meet competitive challenges.
Design Patterns as DevOps Implementation Engineering Framework
As suggested by the business executive, today’s DevOps implementations struggle to deliver the promised business benefits. The reason is, the operational model dimension is ignored in favor of tools and infrastructure.
Another reason is, DevOps is young and is still at the early stage of the technology maturity model; the expertise and engineering methodologies to implement it as what it is—a technology-enabled application delivery model—are lacking. That’s where DevOps transformation design patterns help.
What are DevOps Transformation Design Patterns?
In this article, the term DevOps transformation design patterns refers to general reusable architectures addressing specific DevOps strategy. They’re not necessarily implementable solutions, their primary goal is twofold:
Offer a neutral language for designing and implementing technology-enabled operational models
Offer an engineering approach that reduces design and implementation duration and effort
They’re formalized architecture design and implementation best practices for DevOps and platform-as-a-service (PaaS) services.
Design Pattern as DevOps Implementation Engineering Methodology
To meet its goals—technology-enabled application delivery model—the design pattern is structured around four items including purpose, principles, general solution and implementation techniques.
Purpose: Clarifies the purpose of the design pattern along with the DevOps strategy to address.
Principles: Are best practices formulated as rules and beliefs supposed to guarantee expected DevOps strategy benefits? They’re the determinants of the solutions to implement the DevOps strategy.
General solution: Are blueprints resulting from the application of principles? They summarize the functional and technical architecture of the DevOps platform to implement.
Implementation techniques: What are recommended technical approaches, solutions and tools to implement the general solutions?
Why AWS? Why would it be the catalyst for DevOps value?
AWS-ome Design Patterns That Will Change DevOps Implementation
AWS offers such a rich, varied and easy-to-implement services portfolio that it would be unfortunate not to take advantage of it to modernize IT. The rationale underpinning the value businesses can derive from their DevOps transformation experience with AWS is summarized in the following figure:
Rationale of the DevOps Design Pattern
It’s the belief that:
Certain principles determine the AWS services needed to deliver the application delivery model (ADM) architecture that guarantees value.
The implementation of the ADM using these AWS services results in AWS-based DevOps platforms that tangibly deliver expected benefits.
The Four DevOps Principles that Yield AWS Value
Let’s explore the principles I’ve been using to make CIOs happy with their AWS cloud:
Think holistic, not only IT: Reminds that transformation isn’t only about IT but the entire organization’s value stream from marketing and sales to development and IT operations.
Think cross-functional, not only IT: Reminds that the bottom line is to set up a cross-functional application development environments to accelerate business prioritization, problem-solving and decision-making, and meet time-to-value objectives. They’ll be the core value propositions of the DevOps platform.
Make it agile, flexible, and speedy: Reminds that agility, flexibility and speed are the catalysts for competitiveness and business value. They’ll be part of the DevOps platform.
Think PaaS: Reminds that AWS provides varied services for implementing state-of-the-art CD infrastructure.
Let’s illustrate three AWSome design patterns that’ll change DevOps experience with AWS!
Implement a Startup Mindset with the Software Organization Factory
The purpose is to implement DevOps strategies focused on time-to-market and time-to-value. It’s adapted for tech startups.
To implement the startup mindset, the DevOps four principles for AWS are implemented as follows:
Think holistic and cross-functional, not only IT: Relevant business and IT resources are put together in a single and dedicated service development team supported by appropriate Scrum and XP agile practices.
Make it agile, flexible and speedy: PaaS services building on agile practices and AWS infrastructure-as-code are provided to make application delivery processes agile, flexible and speedy.
Think PaaS: AWS infrastructure as code and DevOps tools are combined to implement a CD infrastructure in the form of a PaaS platform.
The DevOps platform resulting from the principles application is illustrated below:
Software Organization Factory
The “Startup Mindset” platform is structured around three components including the cloud-enabled operational model, PaaS and the CD infrastructure.
The operational model builds on a three-stage application delivery life cycle:
Plan & measure: It mobilizes business users, development, QA and operations staff on business prioritization issues using agile practices such as Scrum product backlog planning or XP’s user stories.
Develop & Test: It mobilizes business users, development, QA and operations staff across the development life cycle using agile techniques such as Scrum iterations, XP’s test-driven development (TDD) and behavior-driven development (BDD).
Release & Deploy: It leverages continuous integration practices to build and test releases prior to deploying to production.
Monitor & Improve: It’s outsourced to keep the service development team on its core business: developing software.
The CD infrastructure combines the followings services:
AWS CodePipeline and CodeDeploy to set up the delivery pipeline.
AWS infrastructure as code including CloudFormation, OpsWorks and Elastic Beanstalk to generate the CD infrastructure.
Summing It Up
The DevOps business benefits are real, yet, it’s young and is still at the early stage of the technology maturity model. The expertise and engineering methodologies needed to implement it as what it is—a technology-enabled application delivery environment—are lacking.
DevOps design patterns such as the software organization factory fill the gaps of today’s transformation practices:
They provide the architect team and the executive team a neutral language to discuss the business, technological and technical challenges and solutions.
By pointing out the agile practices to leverage, they simplify and accelerate the design and deployment of the operational model.
By pointing out the AWS services and features to leverage they facilitate the design and implementation of the CD infrastructure.
Sven Malvik – I’ve written thousand of lines of code based on lies, lies I hoped were the truth. The truth is that we make false assumptions. Assumptions that kill our business. I’ve built software that was too expensive, I’ve built software nobody wanted, and I’ve built software that never saw the light. If you are like any other developer you probably have lied too. But it doesn’t matter. What matters is the application we haven’t built yet and that we will build based on the truth. We need the truth to create value and to survive. This article will show you 4 ways of how DevOps teams create value.
Isn’t it funny that a lot of people, me included, don’t really know the difference between DevOps and Continuous Delivery? Wikipedia tells that both are pretty much overlapping. In order to continuously deliver one must be familiar with DevOps. Anyway, I will tell you what you can do now to start with DevOps and Continuous Delivery. Here are 9½ simple approaches that tell how you can start immediately.
