devopsy.kinsta.cloud

Author: Andrew Davis

  • DevOps for Salesforce: CI/CD Challenges

    DevOps for Salesforce: CI/CD Challenges

    DevOps encompasses a wide variety of technical and social practices for improving the development, delivery and operation of IT systems. Salesforce has become more than a CRM system; it evolved into a full-featured ecosystem and a low-code platform that made development far simpler and more approachable than traditional code-based platforms. Cross-functional teams build on Salesforce to create custom apps and integrations with other enterprises systems without requiring the rigorous approaches that are common with traditional technologies like Java.

    Now, DevOps for Salesforce has emerged to enable teams to build applications faster. And as teams have built more and more customizations onto the Salesforce platform, the level of complexity has exploded. DevOps has emerged as a critical capability for teams to bring this complexity under control.

    The Primary Challenge of Deployments

    The primary challenge that most Salesforce teams face is deployments. Deployments on Salesforce are not just a matter of moving files, they require sophisticated management of thousands of XML files and careful management of dependencies. A number of commercial companies have appeared to help manage the Salesforce development life cycle, and the Salesforce development experience (DX) initiative has unlocked vastly more possibilities for teams to automate existing processes and adopt new workflows.

    The need for more structured and rigorous approaches to Salesforce development has increased the need for teams to learn from traditional software development and DevOps teams and understand how traditional techniques like modularization and continuous integration can be adapted to the Salesforce platform. Technology and knowledge transfer from the broader DevOps community to the Salesforce community remains an active area of exploration.

    Working in Parallel

    It was the adoption of continuous integration by the extreme programming (XP) community in the 1990s that drove intense interest in the need for automated testing, especially unit testing. This allows teams to create exact copies of each other’s work and develop in parallel. Working in parallel allows for far faster innovation since work doesn’t have to be precisely sequenced. 

    But there’s a cost to working in parallel; these workstreams eventually need to be integrated into a cohesive whole. Software projects following a “waterfall” methodology may delay integration until near the end of the project. But these parallel work streams diverge naturally over time, meaning that integration can take months and require enormous amounts of rework.

    Continuous Delivery: A Distinct Practice

    Continuous integration (CI) avoids this issue by merging work early and often (at least daily). “If it hurts, do it more frequently,” is a mantra in the CI world, where the challenge is to resolve conflicts in small batches rather than letting them accumulate.

    CI creates a constant risk of failure for software, since teams working on one part of the system may be making changes to shared components that can cause other teams’ code to fail. Automated testing provides a layer of protection by allowing functionality to be checked and rechecked thousands of times with minimal human intervention. Creating automated tests requires more effort than executing a manual test, but the cost of re-running each test becomes negligible. Thus, automated testing is a perfect solution for high-velocity teams who need fast and ongoing ways to ensure they are not regressing as they build new capabilities. 

    Continuous delivery (CD) is a distinct practice that’s often mentioned in the same breath as CI. Continuous delivery was popularized by the 2010 book of the same name by Jez Humble and Dave Farley. The practice of CD extends the idea of CI by aiming to deploy code to production in small batches on a regular basis; ideally at least daily. Whereas CI addresses the risks of developing in parallel, CD addresses the risks of delaying feedback from end users.

    Software is built to solve specific problems. But each attempt at a solution by developers is a hypothesis that needs to be tested and confirmed. Only the people who actually experience that problem can tell developers whether these solutions are sufficient. So developers must get fast feedback from end users to have confidence in what they’ve built. Teams that neglect this practice are operating on assumptions and optimism; the longer they delay delivery the more the risks of those assumptions compound. Poor adoption and user experience is to be expected when teams don’t practice continuous delivery.

    Increasing Pressure

    CI creates added pressure to test quickly, CD increases that pressure significantly. Delivering small batches of changes reduces the risk of each change and makes them faster and easier to roll back. But by definition, we are changing production systems that may be used by thousands or millions of people each day. A failure caused by CI may waste a few hours of a developer’s time and delay delivery of a change. But a failure caused by CD can prevent thousands of people from doing their jobs, corrupt production data, expose security vulnerabilities and worse.

    When developers integrate their work in CI, the goal is to run a fast set of tests that confirms nothing major has broken. But work being delivered to production needs to go through a battery of comprehensive tests to ensure it meets all functional and structural requirements. In a waterfall-style project, pre-release testing may involve scores of testers working for months. In continuous delivery, teams must be able to execute tests at every level of the system and have robust confidence that no failures have occurred.

    Comprehensive testing requires testing the user interface, testing processes that span multiple systems, loading and confirming data using APIs, security testing and more. Continuous delivery has driven the need for automated testing to new levels and led to entirely new approaches like canary deployments. Canary deployments are a form of shift-right testing where a feature is initially delivered to a small subset of end users to check for issues before rolling it out to everyone.

    The Essence of Lean

    It’s often said that the main driver of success in DevOps is culture. The acronym CALMS is used to summarize the five aspects of DevOps: Culture, automation, lean, measurement and sharing.

    Culture refers primarily to creating a culture of continuous learning and experimentation. The practices described above are not formulaic and require a growth mindset and a culture that prioritizes psychological safety and information flow.

    Automation is the aspect of DevOps most commonly emphasized (including in what’s been written here). Automation enables dramatic efficiencies and is the central miracle of technology, but automation is not a panacea without addressing these other aspects of DevOps.

    Lean refers to the manufacturing practices pioneered in the 1930s at Toyota and further developed and built upon in the late 1980s and 1990s. The essence of lean is to do more with less by understanding and optimizing the value stream by which work is actually performed.

    Measurement refers to the importance of taking a data-driven approach to optimizations. Measurement provides teams with a way to gauge whether process improvements are effective or not.

    Finally, sharing refers to the importance of sharing information, practices and resources between individuals, teams and organizations.

  • Measuring DevOps Performance

    Measuring DevOps Performance

    As IT becomes increasingly central to our organizations, it is increasingly important to improve our ability to deliver innovations efficiently and safely. DevOps is a movement to reimagine the way we deliver software, with an emphasis on delivering value to end users through automation and collaboration. In the midst of complex changes to complex processes, it’s easy to lose sight of the most important point: Our “improvements” must deliver actual improvements. Measuring the performance of a software delivery team is the basic foundation on which you can assess the impact of changes.

    One of the main contributions of the State of DevOps Report has been to focus consistently on the same key metrics year after year. Although the questions in their survey have evolved and new conclusions have emerged over time, the four key metrics used as benchmarks have remained in place:

    1. Lead time (from code committed to code deployed)
    2. Deployment frequency (to production)
    3. Change fail percentage (for production deployments)
    4. Mean time to restore (from a production failure)

    The book “Accelerate” provides a detailed explanation of each of these metrics, and why they were chosen; those points are summarized here.

    The first two of these metrics pertain to innovation, and the fast release of new capabilities. The third and fourth metrics pertain to stability, and the reduction of defects and downtime. As such, these metrics align with the dual goals of DevOps, to “move fast, and not break things.”

    These also align with the two core principles of lean management, derived from the Toyota Production System: “Just in time” and “Stop the line.” “Just in time” is the principle that maximum efficiency comes from reducing waste in the system of work; and that the way to reduce waste is to optimize the system to handle smaller and smaller batches, and to deliver them with increasing speed. “Stop the line” means the system of work is tuned not just to expedite delivery, but also to immediately identify defects to prevent them from being released, thus increasing the quality of the product and reducing the likelihood of production failures.

    Lead time is important because the shorter the lead time, the more quickly feedback can be received on the software, and thus the faster innovation and improvements can be released.  The book “Accelerate” revealed that one challenge in measuring lead time is it consists of two parts: time to develop a feature, and time to deliver it.

    The time to develop a feature begins from the moment a feature is requested, but there are some legitimate reasons why a feature might be deprioritized and remain in a product’s backlog for months or years. There is a high inherent variability in the amount of time it takes to go from feature requested to feature developed. Thus, lead time in the State of DevOps Report focuses on measuring only the time to deliver a feature once it has been developed.

    The software delivery part of the lifecycle is an important part of total lead time, and is also much more consistent. By measuring the lead time from code committed to code deployed, you can begin to experiment with process improvements that will reduce waiting and inefficiency, and thus enable faster feedback.

    Deployment frequency is the frequency of how often code or configuration changes are deployed to production. Deployment frequency is important since it is inversely related to batch size. Teams that deploy to production once per month deploy a larger batch of changes in each deployment than teams who deploy once per week. All changes are not created equal. Within any batch of changes there will be some which are extremely valuable, and others that are almost insignificant.

    Large batch sizes imply that valuable features are waiting in line with all the other changes, thus delaying the delivery of value and benefit. Large batches also increase the risk of deployment failures, and make it much harder to diagnose which of the many changes was responsible if a failure occurs. Teams naturally tend to batch changes together when deployments are painful and tedious. By measuring deployment frequency you can track your team’s progress as you work on making deployments less painful and enabling smaller batch sizes.

    Change fail percentage measures how frequently a deployment to production fails. Failure here means that a deployment causes a system outage or degradation, or requires a subsequent hotfix or rollback. Modern software systems are complex, fast-changing systems, so some amount of failure is inevitable. Traditionally it’s been felt that there’s a trade-off between frequency of changes and stability of systems, but the highly-effective teams identified in the State of DevOps Report are characterized by both a high rate of innovation and a low rate of failures. Measuring failure rate allows the team to track and tune their processes to ensure that their testing processes weed out most failures before they occur.

    Mean time to restore (MTTR) is closely related to the lead time to release features. In effect, teams that can quickly release features can also quickly release patches. Time to restore indicates the amount of time that a production system remains down, in a degraded state, or with non-working functionality. Such incidents are typically stressful situations, and often have financial implications. Resolving such incidents quickly is a key priority for operations teams. Measuring this metric allows your team to set a baseline on time to recover, and to work to resolve incidents with increasing speed.

    In 2018, the State of DevOps Report added a fifth metric, system uptime, which is inversely related to how much time teams spend recovering from failures. The system uptime metric is an important addition for several reasons. First of all, it aligns with the traditional priorities and key performance indicators of sysadmins (the operations team). The number one goal of sysadmins is keeping the lights on or ensuring that systems remain available. The reason for this is simple: the business depends on these systems and when the systems go down, the business goes down. Outages are expensive.

    Tracking system uptime is also central to the discipline of site reliability engineering (SRE). SRE is the evolution of the traditional sysadmin role, expanded to encompass web-scale or cloud-scale systems where one engineer might be responsible for managing 10,000 servers. SRE emerged from Google, who shared their practices in the influential book Site Reliability Engineering. One innovation shared in that book is the concept of an error budget, which is the recognition that there is a trade-off between reliability and innovation, and that there are acceptable levels of downtime.

    According to the Site Reliability Engineering Book, Chapter 3, “Put simply, a user on a 99% reliable smartphone cannot tell the difference between 99.99% and 99.999% service reliability. With this in mind, rather than simply maximizing uptime, Site Reliability Engineering seeks to balance the risk of unavailability with the goals of rapid innovation and efficient service operations, so that users’ overall happiness — with features, service, and performance — is optimized.”

    The State of DevOps Report shows how these five metrics are interrelated (See Figure 1). The timer starts on lead time the moment a developer finishes and commits a feature to version control. How quickly that feature is released depends on the team’s deployment frequency. While frequent deployments are key to fast innovation, they also increase the risk of failures in production. Change fail percentage measures this risk, although frequent small deployments tends to reduce the risk of any given change. If a change fails, the key issue is then the mean time to restore service. The final metric on availability captures the net stability of the production system.

    Figure 1: Correlation between the five metrics.

    How the Five Key Software Delivery and Operations Performance Metrics Tie Together

    Together, these metrics constitute a team’s software delivery performance. The goal of any DevOps initiative should be to improve software delivery performance by strategically developing specific capabilities such as continuous delivery and the use of automated testing.

    How your team measures these capabilities is another challenge. But “Accelerate” makes a compelling argument for the validity of surveys. Automated metrics can be implemented over time, although the mechanism to do this will depend on how you do your deployments. Salesforce production organizations track past deployments, but it’s not currently possible to query those deployments, so you would need to measure deployment frequency (for example) using the tools you use to perform the deployments. Salesforce publishes their own service uptime on Trust, but that gives no indication of whether critical custom services that customers build on Salesforce are in a working state or not.

    Surveys provide a reasonable proxy for these metrics, especially if responses are given by members of the team in different roles. Guidelines for administering such surveys are beyond the scope of this book, but your teams’ honest responses are the most critical factor. Avoid any policies that could incent the team to exaggerate their answers up or down. Never use these surveys to reward or punish; they should be used simply to inform. Allow teams to track their own progress and to challenge themselves to improve for their own benefit and for the benefit of the organization. As it says in the Agile Manifesto “At regular intervals, the team reflects on how to become more effective, then tunes and adjusts its behavior accordingly.”

    Metrics provide a reliable, long-term indicator of how your software delivery team is performing. They open the door for your team to experiment with different approaches and assess their impact using a common standard. The key metrics described here are important because they emphasize end-to-end performance, and thus incent teams to focus on collaboration towards this common goal. Balancing velocity with reliability is critical, thus these metrics should be viewed together, to ensure that one goal is never emphasized at the expense of the other.

  • Why Salesforce?

    Why Salesforce?

    Over the past 20 years, Salesforce has become the fastest growing enterprise software platform, and a market leader in sales, service, marketing, integration and custom application development among other areas. Although there are many reasons why a company might adopt or consider using Salesforce, the overriding value proposition is that it provides tons of capabilities out of the box, releases new capabilities at an ambitious pace and allows for virtually endless customization and innovation.

    Unlike with traditional platforms, there’s no need to manage infrastructure for computing, data, network or security, and no need to invest in generic system administration that adds no value for the organization. Add to this the accessible and supportive community and learning culture based around Salesforce’s Trailhead, which enables rapid onboarding and skill development, and it’s easy to see the attraction of this platform.

    Salesforce’s mantra throughout the 2000’s was “Clicks not Code,” and their logo was the “No Software” image which became embodied as SaaSy, Salesforce’s first mascot. You could be a declarative developer using clicks alone and build a data model, business process automation and user interface using drag-and-drop interfaces. This opened the door to an entire new job specialization: Salesforce admins and app builders—citizen developers who were empowered to directly customize the platform to meet their company’s needs.

    As Salesforce grew, they recognized the need to enable custom coding on their platform and launched Apex Triggers, followed by Apex Classes and Visualforce in 2008. Admins continued to configure rich business capabilities, while low-code developers and professional developers were empowered to create customizations beyond the scope of what could be built declaratively. Initially, the code developed on Salesforce was rudimentary: a few hundred lines here and there to accommodate unusual calculations and data processing. But gradually, companies began to accumulate tens of thousands of lines of custom code, especially as legacy applications were migrated from other systems. Needless to say, the “No Software” motto no longer strictly applies.

    In the decade since the launch of Apex, 150,000 companies have adopted Salesforce and moved a staggering number of legacy systems onto Salesforce—which now processes billions of transactions per day. Many of these transactions are part of the core CRM and customer service applications, but a huge number relate to custom applications, making Salesforce one of the largest Platform as a Service (PaaS) providers.

    How is DevOps Different for Salesforce?

    As mentioned before, Salesforce provides both SaaS (Software as a Service) and a PaaS (Platform as a Service). By contrast, much of the focus elsewhere in the DevOps world is in moving from on-premise infrastructure to using IaaS (Infrastructure as a Service). This diagram shows an illustration of the differences between these four modes, and how they represent progressive simplifications of what companies themselves have to manage.

    Figure 1: On-premise systems require you to manage all of the resources yourself. IaaS, PaaS and SaaS delegate progressively.

    Managing the Salesforce development lifecycle requires a unique skill set and approach compared to most other platforms. You can say the DevOps movement was jointly realized by both developers (working in traditional languages such as Java) and system admins or operators (working on traditional infrastructure such as servers and databases). There are thus a vast range of well-established tools and techniques in the DevOps world for developing and deploying code, and for managing and updating infrastructure. Unfortunately, almost none of them can directly be used for Salesforce.

    To illustrate this, let’s look at how teams would manage infrastructure using AWS, or deploy an application onto Heroku (a PaaS product also owned by Salesforce). Every aspect of AWS infrastructure can be represented using JSON configuration. JSON can be used to define which AWS services are used, which data centers they are running in and how they’re configured. The AWS CLI (command-line interface) can be used in a continuous integration tool such as GitLab CI to automatically deploy updates to AWS infrastructure every time the JSON configuration changes.

    Similarly, Heroku provides a platform to deploy custom application code (in Java, PHP, Ruby, etc.) and to specify which services (such as Postgres databases) are needed to support that application. The Heroku CLI can be used to automatically update the application whenever the codebase changes. Heroku also provides a tool called Pipelines that allows you to visualize and manage the entire development lifecycle without requiring a third party CI tool.

    AWS’s infrastructure as code approach is a delight for sysadmins, since it allows them to track changes and automate updates. Similarly, Heroku removes the vast majority of the setup and dependencies that developers would need to deploy their applications. Heroku provides a true PaaS that is ready to receive custom-coded applications built in a traditional way: from the ground up.

    Although we can say Salesforce’s Lightning Platform is a PaaS, it actually works extremely different from true PaaS systems such as Heroku. The Lightning Platform allows you to write custom server-side or client-side code, but that code can only run on Salesforce. Although it allows you to define custom database tables and fields, that schema cannot be loaded into any database other than Salesforce’s. These customizations are effectively just changes to the configuration of one Salesforce instance. It’s therefore more accurate to say every customization you can make to Salesforce is basically a Salesforce configuration change.

    Salesforce is actually just a big application that happens to allow for infinite customization. But this means the tools used for managing other IaaS and PaaS products cannot be used to customize Salesforce. Fortunately however, the release of Salesforce DX means the techniques and principles used with other technologies can now be ported over to Salesforce. That is the focus of the book Mastering Salesforce DevOps that I’ve recently released.

    DevOps in the Salesforce World

    Moving from on-premises CRM software to Salesforce removes the need for servers and manual software upgrades. Consolidating customer support and online community management onto the CRM platform removes the need to integrate sales, support and community applications. Migrating legacy applications onto the Salesforce platform allows those applications to share data and processes with the rest of the business.

    Because moving to the cloud dramatically simplifies many of the challenges in delivering IT functionality, many Salesforce customers have been able to innovate quickly without having DevOps practices in place. But even without the hassle of managing servers and building software from scratch, the inevitable growth of complexity eventually causes companies to struggle with issues such as organizations getting out of sync and delays in deploying functionality.

    Salesforce solves so many IT headaches, it’s almost easy to overlook the fact that it’s created some new ones.

    Salesforce has unleashed the ability for companies to focus on their core competencies, instead of struggling to provide basic IT services. Salesforce admins and developers are empowered to spend their time directly creating business value. But with multiple Salesforce admins and developers working at companies year after year, more and more Salesforce customers are finding themselves drowning in all of that business value.

    If you have tens of thousands of components, each providing business value, but no systematic way of tracking, managing or deploying them, you now have a new form of business pain and chaos. Far too much of a good thing.

    What is Salesforce DX?

    Salesforce DX (Development eXperience) is an initiative begun by Salesforce in 2016 and launched publicly at Dreamforce 2017. The goal of the initiative is to re-envision the development experience on Salesforce with a focus on how to empower their development community with the tools and processes needed to develop safely and effectively.

    Salesforce DX is a very broad initiative, including several main teams at Salesforce focused on environment management, custom coding, developer tooling, APIs and more. Although the Salesforce DX teams tackle very diverse needs, their main focus (at least initially) is to improve the developer tooling and development lifecycle.

    The tools and capabilities included in Salesforce DX are made available for free to all Salesforce customers. In that sense, Salesforce DX is an important complement to Trailhead, Salesforce’s free, self-paced, gamified, interactive learning platform. Trailhead is also a major investment for the corporation. Together these are intended to support the growth of skilled Salesforce professionals and to make it easier for those workers to build and innovate on the platform. These multi-year strategic initiatives help ensure that talent shortages and worker inefficiency will not be the limiting factor on the company’s aggressive 30% annual growth.

    Summary

    Salesforce is an extremely powerful and versatile platform. But it’s unique in many ways and it hasn’t been easy for professional developers to adapt their tools and techniques to working on this platform.

    Salesforce DX is a major strategic initiative to ensure that industry best practices such as DevOps and modular architecture are possible on the platform. But making something possible doesn’t mean the process is straightforward. It’s still early days for most teams working to enable DevOps for Salesforce. Fortunately, there is an enormous amount of research and experience that can be borrowed from the broader IT industry. Combining a flexible, powerful platform like Salesforce with the speed and reliability of DevOps processes can open up entirely new levels of efficiency and effectiveness for teams.

    — Andrew Davis