Category: Continuous Delivery

What’s it like delivering modern software today? We posed that question to eight DevOps leaders from a diverse range of industries and development organizations. They were all participants in panel discussions I moderated at CloudBees Connect – held virtually this year. Needless to say, there were no short answers to our questions, but here’s a sampling of some key issues they’re focused on.

• Balancing speed with security, compliance and quality. DevOps leaders say they constantly wrestle with getting the best of both worlds. Many err on the side of caution, but also take advantage of automation to embed security safeguards throughout the lifecycle without diminishing speed. “We want to empower our customers and our engineers, but we need to delight our auditors,” says Jennifer Hansen, director of product management and delivery experience at Capital One.
• Standardization versus empowerment. You need both to thrive in today’s environment, panelists say. “The key is to find that sweet spot between maintaining a structure that your organization needs and giving developers the freedom to drive innovation,” says Daniel Ritchie, distinguished engineer at Broadridge Financial Solutions. Some standardization is helpful, and many programs have adopted common patterns and core toolsets to boost efficiency, but not at the expense of best in class tools that the developers want to use most.

Sanmat Jhanjhari, lead DevOps engineer, Nationwide Building Society agrees. “Nationwide believes in empowering people and creating an environment that you have [what we call] accountable freedom…that’s one of the key pillars of [our] leadership framework.”

“We really want them to be involved in the pipeline construction and to tell us what they need rather than for us to describe or define what they should be doing,” says Bob Kelley, director of delivery engineering, IHG.

• Culture not tools. People and culture – more so than tools – are the key to building great software for customers. That’s why continued learning and skills development is a prime focus of Capital One’s Hansen. “We’re trying to create a shared experience and responsibility across everybody on our team.”

• Scaling up. Organizations large and small need headroom to grow. Even smaller companies are preparing for the day they’ll need broad-reaching capabilities. Says Adam Robertson, head of DevOps at Pinger: “We’re building the foundational elements so we’re ready when we reach enterprise scale.”

• Embracing the future. Leaders say they are enthusiastically embracing the latest development approaches and technologies, including microservices architectures and containerization in the cloud.

• Winning over management. Convincing C-suite execs that DevOps is worth the investment can be tough. Jimmy McNamara, software engineering product manager at Fidelity Investments, says the best way to overcome resistance is to “use data to show real outcomes. It’s about articulating a vision of value and demonstrating that in a very tangible way.”

• Driving collaboration. Leaders have adopted different techniques for promoting collaboration among developers. Some have created open source communities within the company, complete with templates and global libraries, to help developers contribute code and allow other teams to benefit from the contributions.

• Move slow so you can move fast. Leaders say the best way to build momentum across teams is to first establish a consensus, so everyone has shared ownership and feels empowered at the same time. That may take time, they say, but if you invest upfront, you’re going to reap returns faster, longer and better later. “Instead of dictating to teams, we need to create consensus up front,” says Aoife Fitzmaurice, architect for enterprise cloud computing at Fidelity Investments. “If a certain choice wasn’t correct, let’s work together to make the right choice.”

• Harvesting best practices. The diversity of practices across engineering teams can be a strength to development organizations, Fidelity’s McNamara says. DevOps can serve as a “central consultancy” that can “harvest their best practices” to empower the whole organization.

What else is on the minds of DevOps leaders today? Listen to both panel discussions – captured in the award-winning CloudBees podcast, DevOps Radio, to dig deeper into the reality of delivering modern software today:

• Episode 73: Live from CloudBees Connect – The Reality of Delivering Modern Software: Part 1 Listen Now

• Episode 74: Live from CloudBees Connect- The Reality of Delivering Modern Software: Part 2 Listen Now

The following blog post is based on a presentation at CloudBees Connect. The full recorded session is available below.

Few industries depend on software more than financial services. Jimmy McNamara, software engineering product manager at Fidelity, should know: He’s worked in the industry as a software developer and project manager for over two decades, most recently for Fidelity Investments, one of the largest investment firms in the world. Fidelity’s brokerage business alone manages multiple trillions of dollars for clients worldwide.

With so much value at stake, McNamara says that Fidelity can’t afford to let its software products lag behind the industry. It’s why the company keeps more than 10,000 developers busy building and maintaining a broad range of business and customer applications. “Fidelity is a huge believer in IT,” he says. “It’s part of our core philosophy and business strategy. It’s something that’s constantly discussed at the highest levels in the company.”

How great software is produced at Fidelity Investments

McNamara’s job is to help Fidelity’s developers produce great software as fast as possible, but also as safely and securely as possible. Doing so requires him to strike a balance between enforcing security through a central platform and allowing developers to use the very best tools available. “Governance and security are non-negotiable,” he says. “But at the same time, we don’t want to get in the way of our developers if they want to leverage the latest technology.”

To gain access to best-in-class development tools, Fidelity increasingly looks to the cloud. “There’s a huge number of new features and a ton of automation that can be achieved by operating in the cloud,” he says. “It’s a game-changer in terms of the ability to improve operational outcomes, realize market opportunities and grow the business.”

To shift the balance toward cloud over time, McNamara prefers to pull rather than push the dev teams. “We establish new capabilities and we make sure those capabilities are better than before, so our developers are naturally drawn to them,” McNamara says. Almost always, developers find the cloud environments easier to use and more conducive to innovation. “Ideally they don’t need to worry about set up,” McNamara says. “They can just focus on innovation and writing code, and it’s a better experience than before.”

Driven by data

To stay on track, McNamara’s team is constantly measuring how well its evolving platform is performing and delivering value to the business. “We’re very focused on data,” McNamara says. “If you follow the data, you can get to where you need to go faster.” The team tracks a range of performance metrics, such as mean-time-to-recovery (MTTR) and how quickly they are providing the features that the business requests.

The data helps his group prioritize new platform-improvement projects and spot trends early to get a jump on emerging problems. For example, his team may decide to add automation features ahead of time to prevent an issue from impacting the business. Moreover, having access to reliable data drives more fruitful discussions with stakeholders about the value of the group’s services. “It moves the discussion beyond opinion to a more accurate, value-based discussion that really drives our services roadmap,” McNamara says.

In the past few years, the company’s embrace of cloud-based platforms has yielded tangible improvements, such as better operational resiliency, much of it driven by the cloud’s inherent elasticity.

What’s more, the move has given developers a richer environment in which to learn and grow. “It’s great to see a team move to cloud-based platforms,” McNamara says. “You can witness the transformation of their skill set. Their whole role in the organization changes because of the new features available in the cloud. In the end, your developers are happier because they get access to more scale, more automation, and more features. Not only that, but their skills become more marketable.” McNamara also knows that having the best tools is key to recruiting and keeping top talent.

To ensure a successful transition, McNamara suggests hiring engineers who are naturally curious and eager to embrace new things. “The key is to bring on really inquisitive people,” he says. “It’s all about people as far as I’m concerned.”

For those who might be reluctant to give up on their traditional on-premise environments, McNamara suggests leading by example. “If you’re a central team or service like us, you should get your stuff out first. Show it off. Demonstrate how you’ve achieved the outcomes you’ve delivered.”

Feature Flags can improve your continuous integration process in a number of ways. This post will focus on how CloudBees Feature Flags can help improve your developer productivity and shorten the feedback loop with the Flag Override view for frontend validation.  It includes Javascript in examples, but the override view can be used in all available platforms.

The flag override view is a lightweight UI element that is very useful for debugging purposes. This override view exposes the current status for each flag set by developers and allows users to override these flags. All this is accomplished from within a handy popup in the UI of your local development instance or other instances used for quality assurance or user acceptance testing.

For organizations, this view improves developer productivity by enabling them to rapidly validate the functionality of their newest code additions in your local environment frontend. To get started, first import the CloudBees Feature Flags SDK to your application:

import Rox from 'rox-browser';

Then, you will want to add the flag override view to your code. For example, this line adds the override view and controls where to display the view:

<button className="btn btn-flags" onClick={() => Rox.showOverrides('bottom left')}>Flags View</button> 

Check out the Browser side Javascript API for a full list of parameters.

Once you have the override view in your local environment, you can toggle any and all frontend flags in your code to validate various code blocks and how they interact together.

Now you can merge your work back to the master branch faster AND with more confidence!

Shorten Feedback Loops with Feature Flags

Quality assurance team(s) without access to the flag override view are typically stuck chasing each feature developer and waiting for them to modify code anytime QA needs to compare new and old functionality. This feedback loop could take hours if the developer is busy or days if the developer is on vacation, not to mention put a damper on developer productivity. With CloudBees Feature Flags and the flag override view, QA can shorten the feedback cycle by having complete freedom to toggle flags in their environment with the click of a button. There is no need to rope in developers to modify code, rebuild, and deploy just to update a flag. Now QA is empowered to work independently, without asking developers to break their focus or wasting precious business cycles.

In conclusion, the flag override view is a handy screen designed for use in local and testing instances of your application frontend. The view helps developers rapidly prototype new functionality so they can get it merged to the mainline codebase faster. The view also helps QA folks work more independently of developer input and with more control of their experimentation all within a shorter feedback loop. Try the flag override view today!

Technical debt is accumulated as unmanaged flags accrue in an application’s code. And the associated marginal gain (i.e. each unmanaged flag added) directly impacts the amount of time and resources needed for testing. This can lead to more serious issues with the affected development, performance/testing and product processes becoming more expensive.

As software teams grow and become more agile, it’s important to ensure a feature flag solution can scale. But it’s imperative that best practices and techniques are followed to avoid the technical debt accumulated by poor flag lifecycle strategy. This post will cover best practices for avoiding technical debt management when using feature flags.

Strategic feature flag planning to avoid technical debt

Sometimes, an ad hoc fix is necessary that can easily be resolved through an additional feature flag. All too often though, scenarios come up where teams feel pressured with time or resource constraints and resort to implementing feature flag gated code without a second thought. While this is somewhat of a normal byproduct of software development, planning is essential and a fundamental precursor to long-term success with feature flags.

The time taken to plan prior to implementation assists in feature flag due diligence. Teams should ask themselves:

• What will the flag be used for?
• What team will use the flag?
• How long will the flag be in use?

The questions were listed in descending order. It’s important to realize this as each question helps filter flags, with the first dividing flags based on its high-level use and the last providing more granular detailed information. This preparation is critical to avoid the hassle of technical debt management later.

How to Name Feature Flags to Avoid Technical Debt

With these questions answered we can now begin naming flags. Companies have style guidelines for writing code, and should append these with its use of feature flags. Before creating a flag, cement the naming convention to be used; this will standardize the practice and help onboard future employees. Like variables, flag names are best left at the intersection of descriptive and terse (i.e. short and sweet).

Some feature flag management solutions provide a namespace. This is the first funnel or way to search flags later on. A namespace should be used to identify how a flag will be used. For example, a set of flags should be defined to the namespace “releaseFlags” allowing for a separate group of “abTestFlags.”

Once a flag has been created with a namespace, a label can denote which teams are using a flag. Since multiple labels can be applied in a flag management UI, it allows multiple teams to associate themselves with each flag.

The description of a created flag provides verbose details, like flag creation date, or necessary information related to a dependency on another flag. This information is the last and most granular bit of the flag’s information.

Best Practices in Flag Lifecycle Management 

Review flags often. Flag management solutions can display and filter which flags are on, off or experiencing no traffic through an interface. But teams must be vigilant about removing flags and have a well-defined clean up process.

Integration with tools like JIRA can help monitor and associate a ticket with the lifespan of each flag. Once a notification is sent that the flag is to be removed (e.g. the A/B test has finished, canary deployment was successful, etc.), the ensuing removal process is easily outlined from previously established naming conventions.

After reviewing a flag’s description, dependencies can be noted and dealt with accordingly. we can review dependencies on all other associated flags. Referring back to the labels will delegate work as the responsible team will be notified to remove the flag from the code base.

And where in the code should they look? Developers turn to the flag’s namespace, although it may exist across multiple tech stacks defined within an application. This is where an additional dashboard should be leveraged, specifically one that displays the environments and code languages a flag has been used in. With all of this information in place, it’s fairly easy to notify the responsible party and search within an IDE for all instances of the flag.

Summary

The following is a guest post written by Carlos Schults.

Being able to disable code in production is a power that many developers aren’t aware of. And that’s a shame. The ability to switch off some portions—or even complete features—of the codebase can dramatically improve the software development process by allowing best practices that can shorten feedback cycles and increase the overall quality.

So, that’s what this post will cover: the mechanisms you can use to perform this switching off, why they’re useful and how to get started. Let’s dig in.

Why Would You Want to Disable Code?

Before we take a deep dive into feature flags, explaining what they are and how they’re implemented, you might be asking: Why would people want to switch off some parts of their codebase? What’s the benefit of doing that?

To answer these questions, we need to go back in time to take a look at how software was developed a couple of decades ago. Time for a history lesson!

The Dark Ages: Integration Hell

Historically, integration has been one of the toughest challenges for teams trying to develop software together.

Picture several teams inside an organization, working separately for several months, each one developing its own feature. While the teams were working in complete isolation, their versions of the application were evolving in different directions. Now they need to converge again into a single, non conflicting version. This is a Herculean task.

That’s what “integration hell” means: the struggle to merge versions of the same application that have been allowed to diverge for too long.

Enter the Solution: Continuous Integration

“If it hurts, do it more often.” What this saying means is that there are problems we postpone solving because doing so is hard. What you often find with these kinds of problems is that solving them more frequently, before they accumulate, is way less painful—or even trivial.

So, how can you make integrations less painful? Integrate more often.

That’s continuous integration (CI) in a nutshell: Have your developers integrate their work with a public shared repository, at the very least once a day. Have a server trigger a build and run the automated test suite every time someone integrates their work. That way, if there are problems, they’re exposed sooner rather than later.

How to Handle Partially Completed Features

One challenge that many teams struggle with in CI is how to deal with features that aren’t complete. If developers are merging their code to the mainline, that means that any developments that take more than one day to complete will have to be split into several parts.

How can you avoid the customer accessing unfinished functionality? There are some trivial scenarios with similarly trivial solutions, but harder scenarios call for a different approach: the ability to switch off a part of the code completely.

Feature Flags to the Rescue

Defining Feature Flags

There are many names for the mechanisms that allow developers to switch a portion of their code off and on. Some call them “feature toggles” or “kill switches.” But “feature flags” is the most popular name, so that’s what we’ll use for the remainder of this post. So, what are feature flags?

Put simply, feature flags are techniques that allow teams to change the behavior of an application without modifying the code. In general, flags are used to prevent users from accessing and using the changes introduced by some piece of code, because they’re not adequate for production yet for a number of reasons.

Disable Code: What Are the Use Cases?

We’ll now cover some of the most common use cases for disabling code in production.

Switching Off Unfinished Features

As you’ve seen, one of the main use cases for feature flags is preventing users from accessing features that aren’t ready for use yet.

That way, programmers developing features that are more complex and take a longer time to complete aren’t prevented from integrating their work often and benefiting from it.

Enabling A/B Testing

The adoption of feature flags enables the use of several valuable practices in the software development process, one of which is A/B testing.

A/B testing is a user experience research technique that consists of comparing two versions of a website or application to decide which one to keep. It entails randomly splitting users into two groups, A and B, and then delivering a different version of the application to each group. One group might receive the current production version, which we call the “control,” whereas the second group would receive the candidate for the new version, called the “treatment.”

The testers then monitor the behavior of both groups and determine which of the versions achieved better results.

Feature flags are a practical way to enable A/B testing because they allow you to quickly and conveniently change between the control and treatment versions of your application.

Enabling Canary Releases

If you deliver the new version of your app to your entire userbase at once, 100 percent of your users will be impacted if the release is bad in some way. What if you could gradually roll out the new version instead? You’d first deploy to a small subset of users, monitoring that group to detect issues. If something went wrong, you could roll it back. If everything looked fine, you could then gradually release the version for larger groups.

That’s a canary release in a nutshell. It’s another powerful technique that feature flags might help with.

Customizing Features According to Users’ Preferences

It’s not uncommon to have to customize your application according to the needs of specific users, and there are several ways in which software teams can accomplish that—some more efficient, and others less so (companies that create separate branches or entire repositories for each client come to mind).

This is another area where feature flags could help, allowing teams to dynamically switch between different versions of the same functionality.

Disable Code in Production 101

How do you go about disabling code? That’s what we’re going to see now, in three increasingly sophisticated phases.

First Stage: The Most Basic Approach

We start with an approach that’s so primitive, it maybe shouldn’t be considered a feature flag at all. Consider the pseudocode below:

calculateAdditionalWorkHours(Employee employee, Date start, Date end) {

// return calculateAdditionalWorkHoursSameOldWay(employee, start, end);

return calculateAdditionalWorkHoursImproved(employee, start, end);

}

In the code above, we’re just commenting out the old version of some method and replacing it with a new version. When we want the older version to be used, we just do the opposite. (Well, I said it was primitive.) This approach lacks one of the most fundamental properties of a feature flag—the ability to change how the application behaves without changing its code.

However, it plants the seed for more sophisticated approaches.

Second Stage: Taking the Decision Out of the Code

The previous approach didn’t allow developers to select the desired version of the feature without changing the code. Fortunately, that’s not so hard to do. First, we introduce a logical variable to determine which version we’re going to use:

calculateAdditionalWorkHours(Employee employee, Date start, Date end) {

    var result = useNewCalculation

? calculateAdditionalWorkHoursImproved(employee, start, end)

: calculateAdditionalWorkHoursSameOldWay(employee, start, end);

    return result;

}

Then, we use some mechanism to be able to assign the value to the variable from an external source. We could use a configuration file:

var useNewCalculation = config[newCalculation];

Passing arguments to the application might be another option. What matters is that we now have the ability to modify how the app behaves from the outside, which is a great step toward “true” feature flagging.

Keep in mind that the code examples you see are all pseudocode. Using your favorite programming language, there’s nothing stopping you from starting with this approach and taking it up a notch. You could, for instance, use classes to represent the features and design patterns (e.g., factories) to avoid if statements.

Stage 3: Full-Fledged Feature Flag Management

The previous approach might be enough when your application has only a small number of flags. But as that number grows, things start to become messy.

First, you have the issue of technical debt. Manually implemented feature flags can create terribly confusing conditional flows in your codebase. That only grows worse with new flags being introduced each day. Additionally, they might make the code harder to understand and navigate, especially for more junior developers, which is an invitation for bugs.

Another problem is that as the number of flags grows, it becomes more and more common to forget to delete old, obsolete ones.

The main problem of a homegrown approach is that it doesn’t give you an easy way to see and manage all of your flags at once. That’s why our third and final stage is a single piece of advice: Instead of rolling out your own feature flags approach, adopt a third-party feature flag management system.

Feature Flags Are a CI/CD Enabler

We’ve covered the mechanisms developers can use to disable portions of their codebase in production without having to touch the code. This capability is powerful and enables techniques such as A/B testing and canary releases, which are all hallmarks of a modern, agile-based software development process.

The names for the techniques might vary—feature flags, feature toggles, feature flipper, and so on. The way in which the techniques are implemented can also vary—from a humble if statement to sophisticated cloud-based solutions.

But no matter what you call them, you can’t overstate the benefit these mechanisms offer. They’re an enabler of Continuous Integration, which is essential for any modern software organization that wants to stay afloat.

Carlos Schults is a .NET software developer with experience in both desktop and web development, and he’s now trying his hand at mobile. He has a passion for writing clean and concise code, and he’s interested in practices that help you improve app health, such as code review, automated testing, and continuous build.