devopsy.kinsta.cloud

Author: Andrew Lau

From the Trough of GenAI Disillusionment – a Bright Spot?
“GenAI: too much spend, too little benefit?”

That’s the question Goldman Sachs asked in an influential report this summer.

It’s safe to say we’ve entered a trough of AI disillusionment. Some companies are laying off staff, while others are being targeted for acquihire (for example Character AI by Google, Inflection by Microsoft, and Adept by Amazon). These factors combined with swings in tech stocks indicate that the market has yet to make up its mind when it comes to valuing AI.

Despite the uncertainty, genAI has gained traction in several sectors, and engineers in particular have embraced AI coding tools like GitHub Copilot. Stack Overflow’s 2024 Developer Survey found that “76% of all respondents are using or are planning to use AI tools in their development process this year.”

With AI coding tools, software companies took the approach of testing first and asking questions later. Those companies are now taking a microscope to their investments and need to justify whether their spending on coding assistants is paying off.

Continued Uncertainty Around Performance and Pricing

Earlier this summer, Jellyfish introduced the Copilot Dashboard to measure the impact of the most widely adopted genAI coding tool. We’ve since gathered data from over 4,000 developers at more than 100 companies, giving us a representative sample of how engineering organizations are using Copilot and what impact it’s having on production. No two teams are the same, but the aggregate data can help engineering and business leaders understand whether they’re getting adequate return on their AI investments.

Without measurement, engineering teams can only gauge the value of their investment in terms of engagement. Are the engineers using Copilot? Sure. But are they faster or more productive? Is the code better, or is that speed coming at the expense of quality?

Pricing for these tools is another potential variable. In October 2023, The Wall Street Journal reported that Microsoft was losing $20 per month per user on its GitHub Copilot product. GenAI coding assistants like Copilot rely on LLMs to drive their service, and the cost of querying those LLMs isn’t cheap! Providers of AI coding tools will inevitably make corrections to their pricing, and software companies will have to decide all over again whether they’re getting value for their investment.

After analyzing aggregate data from Jellyfish’s GitHub Copilot Dashboard, here’s what we found:
- Coding is faster, leading to time savings: Across the board, developers are writing code 23% faster. This impact varies by role. For example, mobile developers and backend developers experience more substantial gains, with coding speeds improving up to 42% and 38%, respectively.
- Coding throughput is increasing: Engineers using Copilot saw a 20% increase in pull requests (PRs) created across all engineering work, meaning, in layman’s terms, that these engineers are pushing more code (and therefore more product) out into the world.
- Junior developers see the largest increase in speed: Across all engineers, Copilot is giving junior developers a 4X boost in coding speed. This implies that Copilot is more useful for automating entry-level tasks and less useful for the more complicated work tackled by senior developers.
Where We Go From Here

We’re in the trough of disillusionment of the genAI hype cycle but there is a bright spot. The technology will continue to develop, and more targeted solutions like AI agents have the potential to disrupt markets more than general AI tools.

Copilot is a transformative tool for software engineering, but organizations must be realistic about its limitations. Coding is only a fraction of what engineers spend their time on in the entire software development life cycle. The first measurements we’re seeing of Copilot adoption show meaningful boosts across different roles and workflows in software organizations. Taken together, these boosts add up to time and cost savings that more than justify the investment now while setting organizations up to realize larger downstream gains in the future.

Regardless of what happens in the months to come, companies can’t afford to experiment without measurement. The “trial and error” phase for genAI is over; we need to verify whether the investments are paying off. Experiment, measure, adjust, repeat.
October 9, 2024
How Visibility Helps Devs Focus on Innovation

As software engineering leaders adjust to a future defined by remote and hybrid work models, they are shifting priorities in the organization of work to compensate for the lack of in-person collaboration. According to a recent study conducted by Jellyfish, the top priority for engineering leaders is ensuring that each member of their team is focused on the highest-priority work for as much time as possible. To achieve this goal, these leaders are using real-time visibility and data-driven decision-making to guide more effective management.

Earlier this year, Jellyfish analyzed the work of more than 23,000 engineers and surveyed hundreds of leaders to better understand the current state of engineering management. The research revealed that engineering leaders are shifting resources away from reactive work—such as responding to customer-reported bugs—to the type of work that directly impacts top-line revenue. Additionally, engineering teams are emphasizing growth and innovation, developing the type of infrastructure that will enable their teams and their software to scale long-term.

Prioritizing Visibility

To shift from reactive to proactive work, engineering leaders must first have visibility into what their teams are doing. How can you allocate your time and resources more effectively if you don’t know how they’re being allocated today? Before work can be prioritized effectively, leaders must understand how much bandwidth their teams have to take on new projects. Additionally, they need data-backed insights in order to align the priorities of the engineering team with those of the company at large.

Visibility is also key to avoiding work that wasn’t part of the plan. According to the study, teams spent 22% of their time on unplanned work in 2021, a 15% increase year-over-year. This trend indicates that engineering leaders may not realize when certain types of work are taking attention away from mission-critical initiatives. When asked to name their biggest challenge heading into 2022, 45% of leaders (by far the largest group) selected “making sure everyone is focused on the highest priority work.”

To gain visibility into their operations, some engineering teams are doubling down on data aggregation and analysis. According to the report, while all organizations tend to devote similar amounts of time to “keeping the lights on” and customer support activities, those that invest in a more robust data organization are able to spend 29% more time on innovation and 48% less time on unplanned work.

Building Through Data and Metrics

The report also found that engineering leaders who’ve embraced data-driven decision-making are seeing more productive, healthy and focused teams. These organizations tend to release code more quickly and foster higher collaboration than their peers. Conversely, organizations that don’t prioritize data-driven decision-making are more likely to experience burnout, decreased productivity and less focus on innovative work.

Here again, the difference between embracing data and ignoring data comes down to technology. Aggregating data and metrics manually can be tedious and challenging. However, emerging tools are able to automatically collect, measure and analyze data to offer insights on what teams are prioritizing. As a result, leaders can more easily align engineering work with business priorities, ensure more predictable development cycles, increase financial responsibility and continuously improve operations.

For every engineering leader, nailing this approach to data will be essential for maintaining efficiency as teams expand. More than 79% of engineering leaders expect to grow the size of their teams in 2022, and respondents named employee onboarding ramp time, virtual onboarding and hiring among the most significant challenges they face.

Despite the numerous challenges engineering teams have endured in the past two years, the industry has adapted and become more collaborative as remote and hybrid work models multiply. For leaders who seek to increase data-driven decision-making, new tools and approaches will result in more intentional engineering work and improved visibility and operations.

June 10, 2022
Debunking Misconceptions About Engineering Management in 2020

2020 will be a year we all remember for so many reasons. It has been a year full of uncertainty and confusion, and also one in which misconceptions abound. In the world of software engineering, we feel this acutely as all eyes are on the engineering team to continue building innovative new products and solutions—maintaining the lifeline of the business—even as we grapple with new realities of universal remote work, new constraints on resources and countless distractions.

Given all of that, it should come as no surprise that go-to-market and other business leadership has grown increasingly concerned with the performance and success of the engineering teams on which they depend. Some of their worries are legitimate, but many are merely misconceptions about the state of software engineering and the efficacy of engineering management. To help foster alignment, the Jellyfish Data Science team investigated by looking at engineering management and team data across dozens of software development teams representing thousands of engineers. Below, three important misconceptions about engineering management in 2020 are debunked.

Misconception 1: Software engineering teams are working less or being less productive amid COVID-19, universal remote work and lack of management oversight

The idea that teams are being less productive is not altogether unfounded. Nor is it surprising given the fact that it’s much more difficult to see what’s being built by the engineering team and how the team is being managed directly. It’s true that among the many teams sampled, productivity decreased across the board about a month or two after the shift to universal remote work (after a spike in productivity initially). However, as things have settled into a new normal, productivity metrics on engineering teams have rebounded and in many cases even increased.

The reality is that engineers are working almost the exact same amount of time as they had been before the pandemic. In fact, the average number of coding days per week has increased 2% (it stands at about 2.2 days). The difference is that engineers are adjusting their working hours to accommodate family schedules. Peak coding hours have shifted four hours earlier, and software engineers are working longer and less regular hours during COVID-19.

Across the board, engineering metrics suggest productivity is up relative to the same period last year. The number of commits has increased, merged pull requests are up more than 10% and the number of issues resolved has increased a whopping 8.7%. The exception here is issue cycle time, which has increased more than 21%. Engineers may be taking on meatier projects, and are working equally, if not more diligently, to ship them.

Misconception 2: Communication on engineering teams is breaking down due to remote work during COVID-19

Again, this worry is not unfounded, especially for older companies where co-located work was important for effective communication and for larger companies where communication is crucial to maintain business continuity across divisions and departments. But among younger, and especially software companies, the move to remote work was not so painful and communication did not break down nearly as much as might have been anticipated given the already widespread adoption of videoconferencing tools such as Zoom or Google Hangouts and messaging platforms such as Slack.

The reality is it’s difficult to say what has happened with communication. Certainly, informal office banter and hallway catch-ups have taken a nosedive. But communication about engineering work—communication that is measurable—has actually increased, possibly enough to fill those gaps. Review and comments on pull requests, for example, have increased by 5% and 4% respectively, and Slack’s CEO reports the number of Slack messages per user per day has increased 20% (which echoes our findings that Slack messages in engineering work channels have increased). The conclusion is that while informal communication may be down, measurable communication—especially around engineering work—has actually increased.

Misconception 3: Engineering management has lost visibility into the work their teams are doing as their teams have gone remote and they can no longer walk the proverbial “floor”

Visibility into the work of engineering teams has not broadly decreased in recent months, but that doesn’t mean visibility is not a problem. Visibility remains a top challenge among engineering leadership.

The reality is engineering management has had poor visibility into their teams’ work long before they went remote. According to the “2020 Engineering Management and Alignment report,” this poor visibility has led to a misunderstanding and misrepresentation at the highest levels of what types of work need to get done and how many resources they’ll require. The report found, for example, that engineering leaders polled overestimated the amount of resources available for road map and new feature work by 62%! Without a more complete understanding of what work is being done, the study asked, how can engineering leaders take corrective actions and ensure the right projects are being prioritized?

But not all is lost in this regard. The study also found that visibility dramatically increases, by 60% on average, on engineering teams that have implemented engineering management platforms. Those teams report 16% less time spent on customer support work and 35% more time available to work on “road map work or strategic customer development—the work that moves the business forward.”

Ensuring alignment with the business is critically important for technical leaders during times of uncertainty. If you can get ahead of these misconceptions, you will better enable your team for success now and in the future. Data-driven engineering leaders already know this to be true when times are good and resources are easier to come by. If you’re not already, strike the misconceptions your business leaders may have by sharing the realities on your team.

November 2, 2020