devopsy.kinsta.cloud

Category: DevOps Practice

4 Mobile Software Testing Use Cases Where Device Clouds Don’t Work

A device cloud, which provides on-demand access to a variety of different types of mobile devices, may seem like the most efficient and effective way to run mobile tests–and it is, in many scenarios.

But depending on which types of mobile tests you need to run or how you need to run them, they aren’t always the best solution. If you blindly turn to device clouds by default to meet your test infrastructure needs for all use cases, you could end up shooting yourself in the foot.

To prove the point, I’d like to walk through four mobile testing use cases where you either fundamentally can’t use a device cloud or where the disadvantages of cloud-based mobile testing outweigh any benefits of the strategy.

What is a Device Cloud?

Before diving into a discussion of mobile testing scenarios where device clouds don’t make sense, let me make clear what I mean by “device cloud.”

I’m referring to a cloud-based mobile testing environment where developers and QA engineers can run software tests on remote devices. In other words, a device cloud is a “cloud” of mobile devices you can access on demand for executing tests. Device clouds are an alternative to running tests on mobile devices that you have to set up locally and manage yourself.

They have become a popular solution for mobile testing because they offer several benefits compared to testing with local devices:

● Testers can easily test applications across hundreds or thousands of different mobile devices without having to purchase and set up each device themselves.
● You typically pay only for the time during which your tests run, making device clouds a more cost-effective solution in many cases than testing on devices that you have to acquire locally.
● Because most providers add new types of mobile devices to their clouds on a regular basis, testing teams always have access to up-to-date mobile hardware.

A variety of vendors (including Kobiton, where I work) offer device clouds that deliver these benefits.

Why Device Clouds Don’t Always Work for Mobile Software Testing

Device clouds are great for use cases where teams need to test mobile apps across many mobile devices and don’t face special requirements. However, depending on your needs and priorities, they may not make sense for your testing routine.

Here are four scenarios where you can’t or shouldn’t use a device cloud for mobile testing.

1. Testing for infrastructure disruptions

Devices hosted in a device cloud operate under best-case conditions. They are fully charged, professionally managed devices that are connected to very stable, high-speed networks.

That’s good in the respect that it means the devices can run tests reliably. But it’s bad for teams who want to understand how their mobile software performs under less-than-ideal circumstances. If you want to know how Bluetooth interference or a spotty cellular connection could impact mobile app performance, for example, you won’t be able to simulate those conditions using a device cloud.

You can simulate these conditions by deploying devices locally in what QA professionals call an isolation chamber, where testers can introduce “chaos” like weak network connections to evaluate how the software responds.

2. Testing mobile apps with strict security requirements

Testing software on a device cloud requires you to upload your applications, along with any data that you need to feed into them for testing purposes, onto mobile devices owned by a third party.

In most cases, this isn’t a big deal. Providers secure their infrastructure and network connections to protect sensitive data.

But if you’re dealing with exceptionally rigid security requirements–as you might in highly regulated industries or in situations where you’re testing apps that manage very sensitive data–a device cloud might create too many security risks. You’re better off in that case testing on-premises, where you have full control over data and software.

3. Minimizing test execution time

Mobile software tests that take place on remote devices hosted in the cloud almost always take longer to complete than tests that run locally. The main reason why is that cloud-based tests require data to move over the Internet, which is slower than local network connections.

To be clear, we’re talking about delays of perhaps only 200 or 300 milliseconds. In most cases, that’s not a big deal. But if you require ultra-fast test execution, which is beneficial in scenarios where you need to run a large number of tests and you don’t want them to delay your software delivery pipeline, cloud-based testing might slow you down too much.

4. Testing on obscure devices

Major device clouds cover most of today’s most popular types of mobile devices, which makes them a great solution if you need to test on devices that are widely used by consumers.

But if you need to ensure that your apps run on more obscure devices–such as specialized IoT devices that you might use in your company’s facilities–you’re less likely to find them in a device cloud. You’ll need to acquire the specialized hardware and run tests on it locally.

Conclusion

Device clouds are a great way to meet garden-variety mobile testing requirements. However, not every mobile testing use case is simple enough to support using a device cloud. Sometimes, tests require a level of security, performance and control that only local mobile test environments can deliver.

October 12, 2023
HashiCorp Streamlines Workflows to Improve Productivity

HashiCorp today launched a series of tools to address everything from streamlining the way infrastructure is managed and applications are delivered to how secrets are kept.

Announced at the HashiConf Global conference, these latest offerings extend the Terraform and Vault portfolio to enable DevOps teams to programmatically automate DevSecOps workflows.

Specific capabilities added to HashiCorp Terraform and Terraform Cloud include a test framework to, for example, reduce misconfigurations along with, in beta, tools for generating tests, tools for launching tests across integrated Terraform modules and a private preview of tools for provisioning infrastructure at scale. In addition, the company is generally making available an enhanced editor for finding and resolving errors in code created using Visual Studio Code.

In terms of costs, Hashicorp is also making generally available an ephemeral workspaces capability to limit the amount of infrastructure resources consumed by Terraform itself.

At the same time, HashiCorp has extended Waypoint on the HashiCorp Cloud Platform (HCP) to define golden paths and workflows for application delivery using templates and add-on modules that make it simpler to identify dependencies. Waypoint was previously announced and is still in beta.

Finally, Hashicorp is adding an alpha edition of HCP Vault Radar, a tool to scan, identify and remediate secrets inadvertently stored in software, in addition to making HCP Vault Secrets generally available and adding a secrets synchronization capability, available in beta for Vault Enterprise and HashiCorp Vault 1.15. HCP Vault Radar is based on technology that HashiCorp gained with the acquisition of BluBracket. HCP Vault Radar will be available in beta in early 2024.

Chris Van Wesep, senior director for product marketing for HashiCorp, said, in general, the company is committed to making it easier to securely manage IT infrastructure and applications at scale across multiple IT environments.

While infrastructure-as-code (IaC) tools such as Terraform have been a boon to productivity, the developers that typically employ these tools have little to no cybersecurity expertise so it’s not uncommon for mistakes to be made. The issue organizations are struggling with today is cybercriminals have become more adept at exploiting those vulnerabilities.

At the same time, secrets management has become a significant issue because many developers will store secrets in plain text to streamline workflows while building an application. Unfortunately, many of these secrets wind up being exposed in a production environment simply because no one remembered to remove them.

It’s not clear to what degree organizations are centralizing the provisioning of IT infrastructure to improve cybersecurity. As forthcoming regulations increasingly require them to lock down their software supply chains, however, it’s only a matter of time before most organizations that build software will need to revisit their DevSecOps workflows.

In the meantime, the hope is those goals can be achieved without adversely impacting developer productivity. The level of cognitive load on developers today is already too high, so the more processes are automated, the less time developers should have to spend managing infrastructure. The challenge, as always, is achieving that goal in a way developers will actually embrace.

October 11, 2023
Microsoft kills Python 3.7 ¦ … and VBScript ¦ Exascaling ARM on Jupiter

Welcome to The Long View—where we peruse the news of the week and strip it to the essentials. Let’s work out what really matters.

This week: VS Code drops support for Python 3.7, Windows drops VBScript, and Europe plans the fastest ARM supercomputer. (more…)

October 11, 2023
Pulumi Adds Internal Developer Portal to Portfolio

Pulumi today added a portal to its portfolio, dubbed Pulumi for Platform Teams, to centrally manage environments through which developers can self-service their own requirements within the context of a set of pre-defined guardrails that ensure compliance and improve security.

In addition, the company also announced general availability of the previously launched Pulumi Deployments orchestration tool for deploying workloads.

Pulumi CEO Joe Duffy said both offerings are part of an ongoing effort to extend the company’s portfolio of offerings beyond infrastructure-as-code (IaC) tools.

Pulumi for Platform Teams is designed to be deployed via the Pulumi Cloud platform for configuring infrastructure, and provides DevOps teams with a service catalog that can be integrated with DevOps workflows via a REST application programming interface (API). That approach makes it possible to leverage hundreds of existing Pulumi CrossGuard policies for automating compliance. In addition, there is a policy-as-code engine to author policies that automatically correct configuration violations.

Pulumi for Platform Teams also includes a plug-in for Backstage, an open source project for building internal developer platforms (IDPs) originally developed by Spotify that is now being advanced under the auspices of the Cloud Native Computing Foundation (CNCF).

With the launch of Pulumi for Platform Teams, the company is providing the building blocks that will enable DevOps teams to build and maintain an IDP that complies with multiple regulatory requirements, noted Duffy.

IDPs are at the core of many efforts to embrace a platform engineering methodology that promises to enable organizations to manage DevOps workflows at scale. The overall goal is to reduce the amount of time developers spend on maintaining environments to enable them to spend more time writing code while at the same time reducing the total cost of DevOps by eliminating redundant tools and processes.

Regardless of approach, given the current overall state of the economy, there is more pressure than ever to improve developer productivity. The goal is to enable developers to self-service their own requirements as much as possible within the context of a standard set of “golden path” workflows. When a developer requests a new environment, the IDP uses calls to application programming interfaces (APIs) to set up a namespace, updates configurations, pulls any images that might be needed and connects to eternal resources such as databases. The IDP should provide developers with enough flexibility to employ new tools when required without unnecessarily adding tools and processes that are redundant to capabilities that already exist.

As every IT leader knows, that’s a fine line to walk, so they would be well-advised to embrace platform engineering in phases. The most critical asset any DevOps team has is culture. Major changes to existing workflows that developers have not bought into are likely to do more to increase friction and strife than to reduce them. When it comes to change management, slow and steady usually wins the race. After all, the worst thing that can happen after embracing platform engineering is for developer productivity to decline rather than improve.

October 11, 2023
The Ultimate Measure of Software Development Effectiveness

There is a famous quote by Peter Drucker that states, “You can’t manage what you can’t measure.” In our industry, measuring how quickly we can get working software into the hands of our customers has become a mantra for agile development teams, and it’s even being talked about in the boardrooms.

Customers measure the value of software by how quickly it can solve their problems. Customer needs change constantly. Just like us, they are responding to the needs of their own customers. Reacting quickly to customer feedback and turning that feedback into working features is key to customer satisfaction and retention.

Having these directives in mind, it’s no wonder our teams are being asked to deliver working software sooner. Often, the main roadblock in achieving this goal lies not in developing the features but in getting them deployed.

Let’s focus on three areas that determine your team’s deployment performance, which, in turn, directly impacts your software development effectiveness.

1. Frequency of Deployments

When I meet developers, I ask them a simple question: How often do you typically commit your code? A typical answer is a few times a day or even a few times per hour. However, if you ask the same person a question about how often their application is deployed to production, the answer varies widely.

Some will say weekly or daily or still – very rarely – even a few times a day, which is software development nirvana. However, a more common answer is monthly or quarterly. I’m also not surprised if someone says that they release annually. They say it with an honest level of pride and accomplishment.

Let’s put these two things – code commit and deployment frequency – into a broader perspective. If the goal of measuring software development effectiveness is getting working software into the hands of the users, it doesn’t matter how often you commit your code but how often you can give your users new value.

Delivering small improvements to your customers has more impact on client satisfaction than giving them a big gift (new release) once a year.

2. The Control of the ‘Launch’ Button

Triggering a deployment includes someone instructing a deployment, pushing a “deploy” button, or committing triggered deployments. How many people in your organization can promote the application to a production environment? If this privilege is limited to a single person or just a handful of people – we have a problem.

There are many reasons you might want to limit access to production. It could be because you don’t have safety measures built into your deployment process, such as role-based access control. You may believe in the delusion that you can protect your organization just by having fewer people with access to the production environment.

The reality is that artificially limiting user access to production significantly reduces your chances of deploying frequently. Imagine if the “person in charge” must take a sick day or is traveling. Remember that access to the production environment is also required for bug fixes when your organization applies a fix-forward versus rollback strategy.

3. The Level of Deployment Automation

Measuring software development performance is directly correlated to the level of process automation. You can’t manually deploy to hundreds or thousands of customers in a reasonable amount of time. The manual process does not scale, as time is the one thing we can stretch.

Ask yourself: Is your deployment process manual, partly automated (where parts of the process are automated, such as scripts that can be triggered to perform application deployment tasks), or can you claim that you fully automated your deployment process where there is no manual intervention, apart from push-button approvals.

The more you automate your deployment process, the more productive your development team can be. For complex deployments, you may need to target many customers, sets of infrastructure or even physical locations. You might need to apply different configurations or hit agreed maintenance windows for customers or regions. One way to accelerate software delivery for these scenarios is tenanted deployments.

Tenanted deployments capture these differences and apply them to a shared deployment process, allowing the complexity to be automated. You can manage tenanted deployments with tags, variables and customizations while reducing the cognitive load for team members.

There are many ways deployment frequency impacts software development effectiveness. When you deploy more often, you reduce deployment risk, increase quality and get feedback sooner. One action you can take to reduce batch sizes is to increase deployment frequency. Smaller batch sizes will also allow you to recover faster when a problem arises.

Remember that the longer the time between deployments, the more changes and points of failure build up.

As you think about ways to increase software development effectiveness, think about the deployment process as a way to get close to the ultimate measure of developer success–getting the working software into the hands of customers sooner!

To hear more about cloud-native topics, join the Cloud Native Computing Foundation and the cloud-native community at KubeCon+CloudNativeCon North America 2023 – November 6-9, 2023.

October 11, 2023
But Are You a Platform?

Funny how our obsession with buzzwords sometimes shoots us in the foot. While I’ve discussed this for specific tech spaces, this time, I’m going to point out a general trend that we’re currently experiencing. I will note up front that neither you nor I can stop it—I’m just here to raise awareness.

Currently, everything IT is a platform, and it smacks of resume padding. Your keycard system is probably “A platform to secure physical and virtual infrastructure across the entire organization, specializing in end-to-end security.” Even though you’re just going to buy it to secure (and, by extension, open) doors.

All joking aside, most of the time, your vendor is going to offer much more than you need, increasingly due to M&A activity, but it’s true for internally developed stuff, too. This works really well for them, because your current need is a gateway drug, and then they can cross-sell you the other 50 functionalities. It works okay for some customers who are looking to simplify their infrastructure and others who are looking for the proverbial “one throat to choke,” but honestly, for most of us, it is a waste of resources across the board.

Ask yourself, “What percentage of this product do I need?” then ask yourself, “How much of this functionality is already implemented in the 15 other platform solutions we have that now overlap with this vendor?” And finally, ask, “What is the possible unexpected overhead of implementing a platform where a tool will do?”

The goal for us is not to provide vendors with recurring revenue; it is to get the job done and keep complexity from forcing us to hire people to manage more of the menagerie.

I have mentioned the current trend of consolidation before—the more consolidation happens, the more your existing vendors will try to solve more of your problems. Sometimes, they will do an excellent job of it and shouldn’t be dismissed just because they’ve become a platform. Other times, it is purely a marketing ploy, and you should do an evaluation of both the market and competitors before jumping on board.

For the most part, I’m preaching to the choir. Most of you have the “What do we need? What do we have available but is underutilized? What do we need that’s new?” system down, but some of you don’t. Think of a toolshed where you have 15 digging tools, but you only dig post holes. You need a post-hole digger in your shed and probably a shovel (because everyone uses a shovel now and then). The rest? In the case of physical tools, you can keep them if you have spare room in the shed but don’t need them. In the case of IT infrastructure, you’re paying for them, paying to maintain them, and/or paying for resources to keep them running. So don’t. Buy what you need, and limit broad solutions to a couple of valued partners or spaces where the intersection of previously separate tech spaces provides value (some app security, for example, is super well-served by the platforming trend because all the formerly separate steps now share data for a more broad view of a given apps’ security that is deeper than current SIEM capabilities).

And keep rocking it. Imagine if you could get down to one of each tool you need and every team was an expert and each of those tools was best of breed … Okay, enough epic fantasy. Just try to minimize what you have to work on/with, and keep serving up the organization professionally to the world and employees.

October 11, 2023
Using DORA Metrics to Measure DevOps Success

The DevOps Research and Assessment (DORA) metrics have fundamentally changed how leaders interact with developers—and, in turn, how developers interact with their code. Although these metrics have changed significantly since their introduction in 2014, their intent remains to provide a numbers-based approach to improving the developer experience.

DORA metrics play a critical role in many development workflows. These metrics posit that four focus areas—deployment frequency, lead time for changes, change failure rate, time to restore service and reliability—can differentiate high-performing software engineering teams (elites) from lower-performing teams (non-elites).

Why is this a meaningful difference? High-performing teams reliably achieve faster time-to-market by removing menial tasks from their workload and expediting the development and deployment processes. Elite teams provide a better customer experience, deploy robust code faster and maintain better availability. And engineering teams that codify their procedures and track using DORA metrics are more likely to achieve these results in a shorter time frame.

Why are DORA Metrics the Baseline for Many Developers?

Most departments have tangible objectives and key results (OKRs) to present to executives. For example, the finance department typically looks to achieve a certain revenue threshold, while the marketing department may report on social media traffic or engagement.

Software engineers have spent years grappling with how to demonstrate success to non-engineering departments. Although it’s relatively easy to identify when a deployment goes wrong, many developers have difficulty translating how these breakdowns occur. Engineering teams cannot generate buy-in for necessary process improvements or tool adoptions without the vernacular to discuss success benchmarks.

DORA metrics provide a language for developers to use when speaking with executives and other teams. Moreover, DORA metrics enable developers to understand their process’ weak points and identify areas for improvement internally.

Making the Most of DORA in 2023

It’s incredibly frustrating to fall short of development goals or hit roadblocks like post-deployment errors and bugs. DORA metrics are the secret to circumventing these challenges. Here’s how DORA can go beyond codification to help your team achieve quicker, more reliable development and deployment:

Accelerate Time to Market

Tracking DORA metrics can significantly curb a team’s time to market—a key competitive advantage in the fast-paced world of software development. Enterprises that deploy more quickly and frequently accrue greater revenue over time than less efficient enterprises.

A streamlined deployment process looks different from organization to organization. That is why DORA metrics are essential; they pinpoint each organization’s specific areas of improvement, enabling DevOps leaders to make informed decisions about remediation. Common indicators for lagging time to market include non-orchestrated deployment processes and manual testing procedures, which distract developers from what they do best: coding.

Leaders looking to isolate issues with time to market should scrutinize their organization’s deployment frequency, as this metric can indicate process inefficiencies. Deploying irregularly? Your team may be batching too many updates, leading to overloaded, riskier deployments. Alternatively, infrequent updates may mean your organization has outgrown manual deployments. If that’s the case, consider adopting continuous integration and deployment (CI/CD) processes—these offerings improve time-to-market through immediate feedback and automated testing. Successful CI/CD implementations are proven to increase daily deployments by over 1000%.

Improve Development Processes

Although highly interconnected, deployment and development delays stem from different issues. Think of it this way: Low-performing engineering teams might deploy frequently—several times a day, even—but without quality assurance, those deployments are likely erroneous and may lead to extended system downtime.

DORA metrics are a powerful way to isolate deployment issues from gaps in the development process. For example, an engineering team that tracks DORA may find their deployment frequency optimal, but their time to restore service and change failure rate lagging. This discrepancy indicates the team needs to improve its testing practices by rigorously reviewing code and managing application dependencies better. Additionally, the team should consider implementing more advantageous deployment and rollback processes, including but not limited to progressive deployments and automated rollbacks.

Remove Menial Tasks From the Workload

Some organizations approach DORA metrics and other OKRs as productivity benchmarks that can be wielded against individual engineers. This perspective has even tinged how many developers view DORA metrics. Yet Google research demonstrates DORA metrics are most successful when adopted by high-trust, low-blame cultures. In these enterprises, DORA metrics reveal organizational opportunities for improvement—never should they reveal lagging individual performers or provide reasons for unnecessary busy work.

In fact, DORA metrics often eradicate rote tasks from developers’ workloads. Teams that use these metrics to identify and implement better incident management processes spend less time fighting fires (e.g., reworking code after a costly manual rollback) and more time doing higher-value work, like building new experiences. Automated processes like automated rollbacks and CI/CD empower developers to improve the quality of individual deployments while also reducing stress

Make DORA Work for Your Engineering Team

DORA metrics aren’t going anywhere soon. While they continue to evolve, like the recent removal of the elite category, they remain the industry standard for engineering OKRs, they provide a robust framework for success and a common language for developers and non-engineering stakeholders to communicate about performance, challenges and improvements in the software development process. DORA eases the collaboration and decision-making processes, creating an environment where leaders can consistently and effectively make data-driven improvements.

But these improvements are only possible when engineers are empowered by beneficial processes and tools. I encourage leaders who implement DORA metrics into their benchmarking to take results seriously. Is your organization’s time to market lagging? Interrogate your current developer workflow—and actively take steps to address issues through developer-first solutions. Doing so will not only improve your organization’s software delivery capabilities but also foster a culture of continuous learning and improvement.

October 10, 2023
Pulumi Previews Tool to Integrate Secrets and Infrastructure Management

Pulumi today previewed a tool that enables DevOps teams to unify the management of environments, secrets and configuration (ESC).

ESC is designed to enable developers to define reusable environments that combine secrets from multiple sources, including Pulumi IaC, AWS KMS, Azure Key Vault, Google Cloud KMS, OpenID Connect (OIDC) Relying Parties, 1Password and HashiCorp Vault. ESC works with any cloud execution context or tool, including Pulumi, Terraform, Cloudflare Workers, GitHub Actions or Docker.

Pulumi CEO Joe Duffy said multiple environments can be defined and composed together to make it simpler to provision cloud environments at scale and eliminate copy-and-paste errors. That approach streamlines a complex process that is fraught with opportunities for mistakes to be made, including inadvertently leaving secrets exposed in an application after it has been deployed in a production environment, he added.

ESC can manage both static and dynamic short-lived secrets, now considered a best practice for secrets management, in a way that can easily be audited, noted Duffy.

ESC can be run independently of Pulumi’s existing infrastructure-as-code (IaC) tool, or it can be integrated with the company’s identity and role-based access control (RBAC) tools. ESC includes deep integration with any SAML IdP, including Azure AD, Microsoft Entra ID, Okta and Google Workspace.

One of the primary reasons cybercriminals target cloud computing environments is because the developers that provision these services have little to no cybersecurity expertise. As a result, the probability a cloud service will be misconfigured is fairly high. As organizations add additional cloud services, it becomes more probable that mistakes will be made as developers programmatically invoke services in an unfamiliar IT environment.

Pulumi ESC is available for free as a public preview today with the intent to eventually offer multiple tiered versions, including a free offering and others with advanced Enterprise and Business Critical capabilities. The ESC client SDKs, CLI and plugins are all open source.

In general, Pulumi has been committed to an open source strategy that clearly defines what capabilities are going to be freely available versus those that can only be found in the other available tiers of service, said Duffy. One of the issues that too many open source software companies encounter is that they can’t generate enough revenue from subscriptions and services to sustain themselves, he noted. Pulumi, in contrast, has been squarely focused on providing additional capabilities that can only be found in its commercial offerings and provide enough value to warrant licensing, said Duffy.

It’s not clear whether organizations are moving to change the way they provision cloud infrastructure and manage secrets. As the amount of scrutiny of software supply chain management increases, it’s now only a matter of time before those processes are, at the very least, reviewed. The challenge, of course, is that when it comes to how those tasks are managed today, there is a lot of cultural inertia to overcome before development teams adapt to a new approach to provisioning infrastructure and managing application secrets.

October 10, 2023
How to Get Developer Productivity Engineering Right

Executives want a clear measurement of developer productivity, but determining it is proving elusive. Because how do you even define software development productivity? Is it the number of points you completed during a sprint? Lines of code or bugs fixed? Features correlated with tangible business outcomes? Or, is it the satisfaction you feel after a day’s job well done? Or is productivity shaped by the social cues you get from your colleagues?

When Brian Houck, principal productivity engineer, Microsoft, asked this question to the thousands of internal developers at Microsoft, he was surprised to see hundreds of different answers across all the above areas. “Developer productivity is many things and means different things to different people,” said Houck.

I recently attended DPE Forum: NYC, which brought together leaders in the burgeoning field of developer productivity engineering (DPE) to share their experiences implementing DPE in practice. Hosted by Gradle on October 3 at the Nasdaq MarketSite building, the event was an eye-opener into the science behind DPE, data-backed internal studies on developer productivity, and what engineering managers are actively doing to improve efficiencies.

Below, I’ll highlight some key takeaways from talks at the DPE Forum to help you get an edge on this emerging practice. We’ll end with some actionable developer productivity tips that you can apply to your own engineering culture.

Reduce Feedback Cycles to Reduce Cognitive Fatigue

According to Hans Dockter, CEO and founder, Gradle Inc., developers see a huge latent potential in optimizing their toolchain. But, although engineering leadership is becoming more open to new DPE initiatives, they tend not to feel as strongly about its potential. Leaders may not see anything wrong with waiting for a ten-minute build, telling developers to “just check your email while you wait.”

Yet, studies have found that the productivity of multitasking teams is far less than that of teams that focus on singular tasks. Furthermore, cognitive control tasks, in which the brain must build new neural pathways, contribute to added mental fatigue. The pervasive ‘multitasking myth’ leads to a superficial perspective that is a blocker to improving the toolchain processes, said Dockter. “When your muscle is exhausted, it’s a biological effect.”

So, how can we reduce cognitive fatigue? Well, Dockter attributes cognitive fatigue and burnout in part to the lack of quick feedback while testing new builds. Code verification failures, flaky tests and failed CI/CD builds can really yank developers out of a ‘flow state.’ And he’s found that when the feedback cycle is longer than ten minutes, you tend to get more task-switching, accelerating cognitive fatigue and decreasing productivity.

The benefit of investing in DPE, he said, is reducing this feedback cycle, thus reducing context switching that contributes to cognitive dissonance. “You cannot have a great developer experience without a great toolchain experience. And that’s what DPE is about.”

Case Study: Decreasing Cycle Time at Netflix

Engineering leaders are actively looking for ways to keep developers in the “inner loop” with fewer interruptions. Netflix, for example, has been investing in ways to improve developer experience around their toolchain, like instigating automated retries for flaky tests. Aubrey Chipman, senior software engineer, Netflix, shared how Netflix Engineering is actively configuring and optimizing their testing processes to reduce this cycle time.

As part of these endeavors, many Netflix projects now use Testcontainers to optimize testing. They also use Gradle’s Develocity for predictive test selection (PTS), which Chipman estimated has saved roughly 30 thousand potential hour savings in a year.

Netflix is also toying with remote test execution (RTE) as a beta offering, which distributes tests across remote agents to execute them faster. However, Chipman acknowledged that parallel executions come with potential limitations, such as network security issues and debugging for different environments.

Build a Systematic Approach to Developer Velocity

Although all teams are special snowflakes, there are definitely common requirements they all share. This is why the idea of an internal developer platform (IDP) has seen greater interest lately. IDP architectures deliver common ‘paved roads’ to internal developers with guardrails to remove friction within the software development life cycle (SDLC).

Amado Gramajo, VP, infrastructure and DevOps engineering, Nasdaq, defined DPE as enhancing developer productivity and happiness through accelerated technologies. And to him, the SDLC really has focused on improving the Ops side in recent years. Now, there is a big opportunity to focus on improving the build and testing aspects of this toolchain.

To distribute common features, Nasdaq adopts a systematic approach to developer velocity, as he called it. They haven’t created a centralized team per se but have developed a shared service available to all developers to help them improve velocity. This global service cuts across otherwise siloed teams, bringing them shared requirements, like resiliency and compliance features. As areas like Kubernetes, SRE and containerization become more normalized, said Gramajo, we need to build the operational controls so things can be automated.

Case Study: Tracking Toolchain Data at GitLab

Some are taking the idea of a systemizing developer velocity quite literally. For instance, Lee Faus, global field CTO, GitLab, described the concept of operating a ‘software factory.’ How can we optimize the supply chain to figure out how to produce artifacts that customers are happy with every single time, just like Oreo cookies?

Well, Faus identified a core issue in webhooks, which he referred to as “the dumpster fire of our toolchain,” he said. Every tool emits different events, which are produced by many actions within the toolchain. However, if this data isn’t properly triggered or received correctly, you could easily have data corruption, causing issues in the testing process.

Therefore, Faus recommended properly handling event data with solutions that enable bi-directionality and avoid creating data silos. Ideally, collecting data related to the software development process should be traversable to help developers pinpoint where problems emerge from. And this root cause analysis should be tied to actionable outcomes for remediation.

Base DPE Descisions on the Data

Although defining productivity is challenging, we have data to back up certain actions leaders can take to increase developers’ perceived productivity and satisfaction. After surveying thousands of internal developers at Microsoft, Houck was able to make some data-backed recommendations.

Formerly Set Aside ‘Focus Time’

The first finding isn’t surprising. Developers like to code. They don’t like things that distract them. Houck’s research found that the amount of time spent coding was positively correlated to developer satisfaction. Engineers who formerly blocked off uninterrupted time to code (especially in the morning) are far more likely to say they are productive. This perceived productivity remains, regardless of whether focus time is chosen or mandated.

Practice Meeting Hygiene

One way to free developers up for more concentrated work is to reduce their time in low-quality meetings. These could be recurring meetings that have outlived their usefulness or ones where the information is not relevant in the first place. Yet, interestingly, not all meetings are bad. For example, Houck’s research shows developers are 2.5x more satisfied with one-on-one syncs with their direct manager.

Allow Flexible Work Arrangements

Houck’s research found that happiness and productivity linearly decline as developers are asked to go into the office more often. The findings are clear—remote workers are more likely to feel more productive than fully on-site workers. Furthermore, the developers who don’t have agency in where they work feel 11% less productive. While there are certainly benefits to in-person brainstorming and team-building, leaders shouldn’t force engineers back into the office for an arbitrary number of days.

Final Thoughts

Measuring developer productivity is a tricky art. Therefore, Houck recommends choosing a few metrics, such as developer satisfaction, the amount of dedicated focus time developers have and the degree of friction within their workflows.

Similarly, Dockter recommended measuring the obstacles and hidden costs in the developer toolchain—obstacles like slow build and testing cycles—that can be readily removed to logically enhance productivity. “The toolchain gives you indications that certain teams are not as mature as others,” he said. Implementing DPE is steady progress, not firefighting, so you’ll probably want to sneak these changes in slowly, he added.

Fitter, happier, more productive. Developer productivity engineering is an emerging practice with a lot of science and opinions surrounding it. But there’s also the human factor to consider. A solid engineering culture will require meeting people halfway and giving them purpose and incentives to retain their interest.

Finally, anyone’s bound to get burned out sometimes with work. At least by minimizing the unnecessary toil in the toolchain, we can encourage a more satisfied developer experience that keeps up high productivity—however you want to define it.

October 10, 2023
Survey Sees Observability Costs Spiraling Out of Control

A survey of 200 DevOps professionals working for organizations that generate $50 to $500 million in revenue found observability costs have become more challenging to control.

Conducted by Wakefield Research on behalf of Edge Delta, a provider of an observability platform, the survey found nearly all respondents (98%) have experienced overages or unexpected spikes in costs at least a few times a year, with 51% seeing overages or unexpected spikes in spending at least monthly. Only 1% of companies claimed their observability costs are not rising.

The most common causes for those spikes in cost are product launches and updates (46%), followed closely by log data being mistakenly included for ingestion (42%).

A full 93% said their leadership team is very or somewhat aware of rising observability costs, with 91% anticipating increased scrutiny to reduce costs in the next 12 months.

As a result, 84% agreed they’re paying more than they should for observability, even when they limit how much log data gets ingested.

Edge Delta CEO Ozan Unlu said it’s clear that while organizations are investing in observability to increase overall resiliency, the cost of that investment is higher than anticipated because the underlying platform being used was not designed to consume, store and analyze data at the level of scale required.

As a result, the survey found DevOps teams are attempting to reduce those costs by limiting log ingestion often or all the time (82%), with 98% also limiting the amount of log data they collect.

Despite those efforts, 83% noted that their decision to ingest or not ingest data has led to a dispute within their company.

Other issues that arise include increased risk or compliance challenges (47%), additional staff time spent preparing data for ingestion (47%), internal tension from not ingesting data for some teams/functions (42%), disruptions to processes dependent on data pipeline (42%), loss of valuable insights and analytics (38%) and failure to detect a production issue or an outage (31%).

On average, log data has grown 5x over the past three years, and nearly a quarter of respondents (22%) experienced a growth rate of 10x or higher. Well over a third (38%) generate between 500GB and 1TB of data daily, while 36% generate over 1TB. Only 15% generate more than 10TB of data daily.

Observability platforms promise to unify logs, metrics and traces in a way that makes it simpler to launch queries to identify the root cause of an issue. In comparison, legacy monitoring tools are designed to enable the IT team to track a pre-determined set of metrics that are typically less granular.

It’s still early days as far as the adoption of observability platforms is concerned, but it’s apparent that as application environments grow and become more complex, the need for observability is becoming more acute.

The rate at which DevOps teams will embrace observability will naturally vary, but the biggest obstacle might not be the platforms themselves. Instead, the issue is understanding what queries can help DevOps teams better understand the root cause of an IT issue before there is a major disruption. In the long term, it’s expected that machine learning algorithms will leverage the data collected by observability platforms to automatically identify issues that might lead to a disruption long before it actually occurs.

In the meantime, the rate at which IT environments are becoming more complex is clearly outpacing the amount of budget dollars being allocated to manage it.

October 9, 2023