devopsy.kinsta.cloud

Author: James Smith

  • Do’s and Don’ts of Phased Rollouts Software Delivery

    Do’s and Don’ts of Phased Rollouts Software Delivery

    When it comes to app stability and user experience, the “move fast and break things” approach is a recipe for disaster. To protect brand reputation and to stay ahead of evolving needs and customer expectations, leading Silicon Valley companies’ software engineering teams have adopted phased rollouts, a software development method that involves the gradual implementation of a new feature to a smaller cohort of users at one time rather than all users at once.

    Phased rollouts help companies maintain the highest-quality customer experience (CX) for their users while making changes and software updates. When significant features and updates aren’t rolled out to production all at once, the organization’s software engineering team doesn’t have to deal with all the potential implementation issues at once. With phased rollouts, engineering teams catch issues early in the process so they can quickly fix them and resume the rollout of the experiment, which prevents issues from impacting millions of users down the road.

    While companies everywhere try to emulate the app development techniques of these Silicon Valley tech leaders, many are unable to achieve successful phased rollouts at scale. What they often don’t realize is that before even implementing a phased rollout approach—whether it be to 5%, 10% or 15% of their user base—there are a few steps that must be taken to ensure success. Below, I’ll walk you through the do’s and don’ts of a successful phased rollout plan.

    Do: Release Stability Management

    Companies must drive consistently stable app and web experiences in order to succeed in the market today. Therefore, it’s critical to continually monitor the stability of releases throughout development and testing to catch problems early and decide when to roll features out to users. By measuring and monitoring app stability in real-time, release managers inject accountability, predictability and complete peace of mind into the organization. Taking control of release stability management helps engineers deliver stable applications to drive world-class digital experiences.

    On top of that, software development teams must constantly improve mobile and web apps with new features and capabilities that better enable users to accomplish what they want to do. These kinds of improvements give companies the competitive edge they need to survive in today’s competitive business environment. In addition, monitoring release stability provides engineering teams with real-time visibility into every issue impacting application stability and CX. This level of visibility is critical for quickly diagnosing and understanding the root cause of errors so that engineers can roll out fixes before users are negatively impacted.

    Do: Testing and Early Access

    For a phased rollout, engineering teams must run A/B tests and quality assurance (QA) testing. A/B testing (also referred to as split testing or bucket testing) is a method of comparing two versions of a web page or app against each other to determine which one performs better. This includes trying out features like different screenshots, videos and even different app icons to measure which format users like the most.

    QA testing ensures that the organization delivers the best products or services possible by detecting usability issues that might not be technical. It tests for any problems that impact the end-user experience, from poor navigation to slow load times or unclear web copy.

    With phased rollouts, you must also provide early access to new features for select users to collect feedback and stamp out critical issues before launch. Many software products collect user feedback by including a small checkbox somewhere asking users to help make the software better by sending insights into how they use the application. In addition, social media has become an undeniable avenue for collecting candid user feedback. We’ve all seen it—when a platform changes its interface for certain users before others, users will post on social media sites about whether they are pleased with the new update or not. Historically, there has been backlash when significant changes are made and the company then has to decide whether to keep, change or cancel the new changes to the platform.

    As mentioned above, with an app stability monitoring tool, you can see if new features or experiments are introducing new stability issues and make decisions about pulling back or rolling out to a larger subset of users. By conducting testing and allowing early access to only a specific cohort of users, software engineering teams can balance release speed with application stability to create the best experience possible.

    Do: Allow for A Staging Period Before Going into Production

    As you launch new features, you’ll want to test them in a production-like environment after they go through QA testing. This is called staging, and it has become a popular concept with Silicon Valley companies. Staging is a near-exact replica of a production environment for software testing. Staging is meant to test code, builds and updates to ensure quality under a production-like environment before application deployment.

    A production-like environment is similar to when the app is released into production for users, but it’s not public yet. It provides an accurate view of how the update or new feature will operate and look in a production environment to avoid glitches that are only noticeable in production. Staging can include manual tests or, in some cases, engineering teams use automated software testing, which is more costly to implement and may only be attainable for larger enterprises.

    In this phase, software engineers should conduct additional tests and experiment with the new features to ensure that these capabilities—when released to production or to the larger user base—will not compromise the app’s stability. The last thing you want is for a new feature to break the user experience.

    Don’t: Try to Fix Every Bug or Error as it Occurs

    Since engineering teams often don’t have the bandwidth to fix every coding error that should arise, they need to prioritize and fix the errors that impact customers directly in order to protect revenue. Although organizations should have visibility into every error or bug, not all of them are worth fixing. This causes the signal-to-noise ratio that plagues developer efficiency.

    This is where app stability management plays an essential role. However, many of the app stability monitoring solutions on the market today can help with identifying errors, but leave it up to development teams to figure out which ones need to be prioritized. Engineering teams have to quickly determine which errors are the most urgent to fix, so they need to have reliable tools such as an app stability and error monitoring platform to provide them with visibility to make these important decisions in real-time.

    Phased Rollouts for Progressive Delivery

    Phased rollouts allow software engineers to continuously optimize the digital experience, without impacting the whole user base and negatively affecting the brand. Once mastered, phased rollouts can be used as part of a cutting-edge progressive delivery approach to software engineering and development.

    A progressive delivery approach includes shipping code (sometimes unfinished code) to a production environment multiple times a day, exceeding many companies’ risk tolerance. In order to do this successfully, engineers must have control over who can ship releases and when, and which users can access certain features and updates at what time. Phased rollouts provide a way to shield the majority of customers from the failures that occur at those rapid release speeds. To ship new software releases faster, engineers have to spend less time on QA and testing, but that doesn’t mean they should be overlooked completely. Then, by delivering the new release to a small subset of users first, you can manage risks more effectively and iterate accordingly.

    It’s important to call out that if the engineering team is making minute changes that no users will see or care about, you don’t need to use a phased rollout approach. That said, if it’s a significant cosmetic or operational change, such as updates to the UI, then phased rollouts are crucial. A phased rollout strategy is key for making software or app updates at scale that are necessary for a successful launch in today’s highly competitive market.

  • Code Ownership Is Key to Accelerate Debugging

    Code Ownership Is Key to Accelerate Debugging

    App stability is a fundamental part of every app experience. Broadly speaking, app stability is a measurement of the number of total app sessions that are crash-free or the percentage of daily active users who do not experience an error. End users have little patience for apps that crash or fail. Every application, whether it’s B2B- or B2C-oriented, suffers some level of instability and fails occasionally. Ideally, organizations should aim for app stability scores close to 99.999%. To achieve that quality of performance, software engineers must be fast and efficient in fixing coding errors that are frequently the cause of app crashes.

    But that’s a major challenge using the traditional approach to debugging. Most major applications have a wide variety of engineers, including completely separate engineering teams, all working from a single code base. Each team typically uses their own unique issue tracker to monitor errors, such as Jira or Asana. In addition, teams typically use their own channel in a collaboration platform to communicate internally, like Slack or Microsoft Teams. This causes silos between the various engineering teams, limiting communication and visibility between each team.

    When an error occurs, all the engineering teams are alerted about the bug through their own issue trackers. From there, they figure out where the error occurred in the code and who is specifically responsible for fixing it. This requires each team to manually review code as they try to locate the bug, which wastes precious time as engineers duplicate one another’s efforts. Because teams are using different communication channels, there’s little to no collaboration or alignment between the various engineering teams during this process. Overall, this approach to debugging is a cumbersome and inefficient undertaking that slows the process of repairing apps, ultimately hurting the end user experience.

    Consider a real-world example. Suppose your team is one of a dozen engineering teams, all working together on an application that creates a mobile game. Perhaps your team works on maintaining the in-app purchases feature set, while another works on the account setup flow and yet another team works on character customization features. If a bug suddenly crashes the app for a significant portion of users (resulting in a clear drop in the app stability score), every team will receive an error alert through their issue tracker notifying them that a coding issue has caused the application to fail. No one will know if their team’s code was responsible. As a result, each team is forced to go through the laborious process outlined above. If it turns out that the bug was in the character customization code, then the teams responsible for in-app purchases and account setup have wasted valuable time and resources searching for the error.

    The only team that should be notified of a bug is the one responsible for the part of the code a bug originates from. The emerging concept of code ownership makes this possible, providing a faster and easier approach to debugging. With this method, organizations create code owner files to determine which engineers or teams are responsible for each section of code, feature flags or experiments. If section X within the code fails, then team X will be responsible for addressing it. In the example above, only the team responsible for the code associated with character customization would need to find and fix the bug.

    This makes it much easier to respond to application issues and allocate the correct resources to remedy those issues. For end users, code ownership means that malfunctioning apps will be fixed faster, improving the customer experience and boosting business metrics. From an internal POV, engineering teams become far more cost- and time-efficient, freeing developers to take on new projects they actually enjoy working on.

    A culture of code ownership also helps engineering teams adopt other best practices for debugging. One major example is error prioritization. Not all bugs cause significant app instability and need to be fixed right away. When a team can ignore errors unrelated to their code, they have more time to fully hone in on and evaluate the errors they are responsible for. With this focused approach, they can establish metrics determining if they need to fix a bug immediately or if it can wait.

    Simply put, code ownership eliminates the confusion and guesswork associated with the process of debugging, allowing engineering teams to easily identify, own, prioritize and remedy bugs. Embracing code ownership is critical for companies to improve app stability, which is the most fundamental measure of an app’s health. This ties directly into business outcomes, because app stability is closely associated with customer retention, customer growth, sales and revenue. By speeding up debugging efforts, code ownership has a pronounced, positive impact on stability, boosting each of those metrics.

  • 5 Reasons Why Your App Needs an Error Monitoring System

    5 Reasons Why Your App Needs an Error Monitoring System

    All software applications have errors. Bugs are simply part of software development. That’s why engineering organizations need to know exactly what errors are occurring so decisions can be made around which bugs to fix.

    The fastest way to identify and fix application errors is to implement error monitoring software. However, not all solutions are created equal, and the best error monitoring systems do more than error alerting. They provide insights into the stability of the software, give developers metrics for addressing technical debt and enable better decision making around building features versus fixing bugs.

    The result? Happier developers who build better apps that match customer expectations.

    Here are five specific reasons why engineering organizations should adopt an error monitoring system.

    Error Monitoring Improves Application Health

    At its core, an error monitoring system addresses the challenge to find, fix and prioritize software bugs. However, what makes an error monitoring solution stand out is stability scores, which make it easier to maintain a healthy app.

    • Calculated with real-time error rates and session data, stability scores reflect the percentage of successful user interactions per release. 
    • They tell the real story about the impact of errors and technical debt and provide a single, unifying metric that speaks directly to the overall application health.
    • Stability scores can be used across teams to set stability targets (critical stability and target stability), which in turn can determine whether the app is healthy enough for building new features, or if it’s time to fix bugs.

    An Error Monitoring Solution Increases Team Productivity

    With a lack of detailed information about the source and factors contributing to a bug or crash, developers must conduct research to pinpoint errors, which is a time-intensive process.

    Without Error Monitoring With Error Monitoring
    Sixty percent of developer time wasted investigating and fixing errors, often in code they didn’t write. Forty percent increase in developer productivity with the exact location and source of the error provided by error monitoring solution.

    From detection to resolution, error monitoring with stability scores determines whether debugging needs to be part of the workload and which errors should be addressed first. In addition, the best error monitoring tools help prioritize business-critical and revenue-impacting errors to ensure alignment from top to bottom within an organization.

    Understanding Your Application Errors Can Reduce Costs and Increase Revenue

    With an error monitoring service that catches errors quickly, engineers save time, avoid critical errors and deliver crash-free applications that users want to spend money on. New features can also be rolled out on time with fewer errors, which makes developers and users happy. 

    • On average, companies lose $1,500 per engineer each month over debugging.
    • Knight Capital Group lost $440 million in a mere 30 minutes due to a software bug.

    Technical issues are the number one reason apps get deleted. Thankfully, companies that deliver smooth app experiences benefit from loyal users who trust and spend money on their brand.

    Error Monitoring Retains Development Talent and Boosts Morale

    Technical debt is hard on engineers, both from a productivity and emotional perspective. Without a common language to explain its impact to other teams, developers often become frustrated at bearing the burden of technical debt, which can cause disengagement and encourage developers to leave.

    Error monitoring management and stability scores bring complete visibility and cross-team agreement around software stability and prioritization of errors. Stability scores provide:

    • A common vernacular for discussion between engineering, product, QA and release teams about when it’s time to fix bugs versus build new features.
    • A metrics-driven reflection of the impact of technical debt on the product and clear demonstration about when it’s time to address it.
    • A singular goal for application stability, which turns development and product teams into allies and increases morale across the organization.

    Error Handling Impacts User Satisfaction and Usage, and Safeguards App Store Ratings

    Apps play an important role in our personal and professional lives. Widespread app adoption and daily usage point to why application errors are becoming unacceptable to users, who aren’t shy about speaking up when errors do occur. 

    • Ninety- six percent of mobile app users write bad reviews for sub-par apps.
    • Forty-two percent of 0-star app ratings in the Google Play Store mention app stability and bugs.

    The best way to ensure users have a positive customer experience is to use error monitoring to catch errors in real-time so you can respond immediately. Error monitoring provides a hope for avoiding poor app store ratings and for keeping customers satisfied.

  • APM and Application Stability: Where Two Monitoring Roads Merge and Diverge

    APM and Application Stability: Where Two Monitoring Roads Merge and Diverge

    In software development, application performance management (APM) is one of the grown-ups in the room. Not only has APM been around for a long time, but its solutions have evolved over several generations, making it one of the more mature product categories. 

    APM’s longevity makes perfect sense when you consider its fundamental purpose, which is to provide organizations with a way to understand the performance characteristics of their software. Value is delivered by alerting infrastructure teams when applications are performing slowly or poorly, and APM has the vast experience to do this job well. 

    If APM is one of the old-timers in software development, then application stability is the new kid in town. With the rise of mobile apps and iterative development releases, application stability has answered the widespread need to monitor applications in a new way, shifting the focus from servers and networks to the customer experience. 

    The emergence of application stability has caused some consternation for die-hard APM fans. However, these two solutions embody very distinct monitoring focuses, which leads me to believe there’s room for both tools, as well as different teams for both.

    APM: The Engine for Infrastructure and DevOps Teams

    Before the cloud, organizations supplied their own physical hardware and monitored components such as RAM, disk space, CPU and memory. If you ran out of any of these resources, you were screwed. 

    And that was the beauty of APM: It enabled the people running the applications to anticipate when they’d need more resources. 

    Of course, rather than monitor physical machines, today’s infrastructure, SRE and DevOps teams monitor cloud instances. Gone are the days of running out to Best Buy and lugging back new hardware. Instead, a simple request for a new instance is submitted to a cloud provider, and instant access is received. 

    Rather than make APM obsolete, as some feared, the cloud presents two good reasons for continued reliance on this tool.

    1. Apps Can Be Resource Hogs: A cloud instance is simply a slice of someone else’s computer, which means you still need to know when you’re close to running out of resources on this virtual machine. In fact, extra care is needed these days because you’re likely to run out of space more quickly.
    2. Money Still Doesn’t Grow on Trees: The ease of access to infinite cloud resources means that software companies may fall into the habit of throwing money at problems rather than figuring out how to streamline usage and costs through better efficiency. Companies can end up paying a ton of money to run apps if they continuously spin up more cloud instances without thorough consideration of the cost and need.

    And that’s where APM steps in. Its main purpose is to help infrastructure teams figure out capacity planning. With the cloud, questions shift to include:

    • How can I better manage my cloud instances and optimize usage? 
    • How can I make sure I don’t give ludicrous amounts of money to cloud providers? 
    • How can we tweak and optimize the cloud to reduce costs?

    These are the types of things that APM does extremely well. Anytime you need to figure out how to optimize your current resources or when to buy a new server or cloud instance, APM tells you. This information is invaluable to the people running the applications, namely the infrastructure or DevOps team.

    Application Stability: Where the Tires Meet the Road for Engineering Teams

    Now, let’s be honest, APM doesn’t have its strongest showing when the person who runs the app is also the person who builds the app. And that’s exactly the scenario presented by mobile development and iterative coding. 

    Rather than going through long development cycles, software now gets pushed to the web on a daily basis, and mobile apps tend to have weekly or biweekly release cycles. This release speed is not only encouraged but expected in agile software development.

    That means the chasm between building and running apps has shrunk, especially for mobile apps. More often than not, the people building the apps are the people releasing them. There is no gap. And, with mobile, companies no longer need to worry about expensive physical hardware but rather about the end user experience. The customer becomes the focus.

    Unfortunately, APM is perhaps a little too set in its ways to help you do that. But it’s not APM’s fault. These requirements are not what APM is inherently built to do. It’s great at detecting problems and alerting infrastructure teams so they can toss the issue over the fence to the development team. However, its core strength isn’t providing information on how to fix the problems because it wasn’t built with the developer audience in mind. 

    Trying to refactor APM to help with stability and error issues is like tuning your engine when you have flat tires. They are completely separate components of the car, built for different purposes. You can tune the engine all you want, but it isn’t going to move the car unless you focus on why there isn’t air in the tires.

    In today’s iterative world, development teams care a lot more about how apps are running. There’s a demand for fixing actionable items. Developers want to know exactly what’s broken, what to fix right now and what can wait. In short, developers want aggregation and automation of errors. They want to know, “Do we build or fix?” 

    This trade-off between building new features versus fixing bugs is one of the key factors behind the adoption of application stability management tools. Developers need answers to several questions:

    • Where are the errors in the code?
    • How can we get to those bugs and fix them as soon as possible?
    • How can we tie bug fixing into our planning for the week? 

    Whether running sprints or building in an agile manner, development work is planned in advance. Developers want to figure out how much new feature work will be included in a sprint versus how much bug fixing is required. And they need a tool that helps them automate the answer to that question.

    Benefits of Application Stability

    The beauty of application stability is that it brings together the errors captured by APM and enables developers to see at a glance which ones are worth fixing. As a result, five major benefits arise.

    1. Increased Efficiency: Companies eliminate the problem of infrastructure teams tossing issues over the fence to development teams. Valuable time is saved because application stability tools remove the game of telephone between the two teams and deliver bugs directly to the team that will fix them.
    2. Stronger CSAT: The time to fix bugs goes down dramatically when the person who wrote the code fixes the code. With diagnostic information in hand from the application stability tool, software engineers innately understand what the code does, what the bug means and how to fix it. Faster resolution of bugs that impact the end user experience means that customer satisfaction levels (CSAT) are less likely to drop.
    3. Error Prioritization: Application stability tools group bugs by root cause, making it easy for developers to get a sense of severity at a glance. It’s much easier to determine what to fix first when developers can see which errors are most costly, which affect the most customers, and which bug is impacting a key customer.
    4. Tool Synchronization: Taking it one step further, application stability tools are tied into project management suites. Bugs map directly to tickets created in Jira (or whatever tool is used), and tickets update automatically as priority changes.
    5. Stability Scores by Release: Application stability enables product and development teams to see stability scores by release. Since it’s common to have multiple app versions live at the same time, especially with mobile apps (where DevOps isn’t really involved), companies can’t rely on a single stability score. Teams need to see stability by release so that it’s clear exactly where the errors are and what impact they’re having on users.

    What Percentage of Your Development Team Has a Login to Your APM?

    I’m often asked whether I think application stability will replace APM, and my answer is simple: no, I don’t. APM remains an essential part of developing software, and organizations still need to understand when they’re about to run out of resources and when there’s poor performance. 

    Instead, I see these two solutions co-existing as adjacent categories but helping different teams. Application stability delivers prioritized errors to developers for fixing, while APM works well for enabling ops teams to raise red flags on high error rates and reduce cloud spend.

    Some of you may be thinking to yourself, “Well, my APM product does what you’re describing for application stability, so I’m sure my developers are fine using it.” To which I poise the following challenge: What percentage of your dev team has a login to your APM? What percentage logs in on a daily basis? And, if they do use it, do your developers like it?

    The answers to these questions may surprise you. After all, APM wasn’t really built for developers or for keeping end users happy. In contrast, application stability was born at the customer layer and is designed specifically to monitor the front end and ensure strong customer experiences with web and mobile apps. 

    Once you’ve had a chance to hear from your dev team, it wouldn’t surprise me if you discover that they’re pretty excited about the new kid in town.

    — James Smith