devopsy.kinsta.cloud

Tag: operations

  • Overcoming the People, Culture Barriers to DevOps Adoption

    Overcoming the People, Culture Barriers to DevOps Adoption

    The fact you’re reading this article on devopsy.kinsta.cloud suggests you’re already a DevOps fan. Or maybe you’re visiting this site because you’re considering moving to DevOps in your organization, or you’ve been told you must move to DevOps and are figuring out how on earth to do it.

    Whatever your view on DevOps, let’s get my position clear from the outset: I am a big believer in the power of DevOps to deliver continual innovation to an organization. I advocate DevOps in every project I do because I believe those organizations that innovate and change the fastest become the leaders in their sector, and since almost every business innovation relies on a technology innovation first (“every company is becoming a technology company,” after all), DevOps is the best way for an organization to adapt to, and benefit from, new technologies and gain a lead in the market.

    However, as anyone who has managed the transition from waterfall to DevOps will know—or, in fact, anyone who has overseen any change management project will know—the transition is rarely a smooth one. So, what is behind this reluctance to innovate and work in new ways?

    First, the Technical Arguments

    At some point, most organizations have been burned due to a lack of quality control in technology, whether a security breach, bugs or implementations/changes that went horribly wrong. We all know stories like these. They can have an immediate and catastrophic effect on the organization’s ability to do business. Therefore, there has been a tendency to inhibit change in favor of “stability” and known entities, and to adopt a “Better the devil you know,” strategy.

    DevOps has rocked this view of the world, and teams are reticent to move away from tried and tested approaches and processes because they are held responsible for operational stability. In addition, those people in a quality-control role believe their very purpose and job is under threat. There is also a view that DevOps is system- or application-centric, and can introduce risk to an interconnected IT estate.

    While these are all valid concerns, it is worth bearing in mind they are all technical arguments against DevOps, and they have all been countered before. Implemented correctly, DevOps does not jeopardize stability or undermine quality (in fact, many would argue it does the exact opposite), and it does not undermine a person’s role in development or operations, but rather liberates them from their silos.

    Culture is the Real Reason

    From my experience, innovation of any kind cannot be held back by technicalities alone. While the technical arguments are certainly valid, in most instances, the reluctance to adopt DevOps comes from something much more basic: the organizational culture borne from natural human behavior. Most people are simply risk-averse and resistant to change, so the resistance to DevOps is just a manifestation of human nature.

    The only way for DevOps to succeed is to change the organizational culture to be more open to change and to adopt an “innovation mindset.” The change must also encompass the whole organization. If you only bring half of the business with you, the innovators will become frustrated and find ways to circumvent processes and policy, resulting in additional operational risk and shadow IT.

    The influence that senior managers can have in facilitating—or undermining—the move to an innovation mindset must also be factored in. It is common for senior management to inadvertently foster a hero culture, where those individuals who get things back on track after a catastrophic event are rewarded, yet the well-structured and “stable” DevOps deployments are not given the same weight or focus. Again, it is human nature to be drawn to the dramatic event than the non-event. But you must ask yourself: As exciting as firefighting is, is it a sustainable way to run a business?

    Instilling a new mindset into a business is the same as any other transformation project. It requires you to tackle the four elements of process, organization, technology and information to bring about meaningful and sustainable change. To bring DevOps into your organization you must:

    • Align operational teams with development teams, embedding where possible for early involvement of Ops.
    • Ensure that operational/non-functional requirements have the same weight and priority as functional requirements—no user story is complete without a stable and secure operational environment.
    • Automate where possible to reduce human error and increase the speed of delivery.
    • Improve knowledge management to reduce “hero culture” or bottlenecks where knowledge is concentrated across a few people or one team. Never make exceptions for “heroes”; you need their support more than anyone else because they hold the biggest sway over the rest of the team.
    • Have accurate and comprehensive configuration information. Having a good understanding of the IT estate (including interconnectivity) can help move an organization from a one-size-fits-all governance model to a tailored risk assessment approach through better understanding of the impact of changes.

    DevOps often fails when people can revert to their default behavior. Only by actively identifying and supporting a project in a DevOps manner (considering process, organization, technology and information as previously explained) will you be able to change to a new “business-as-usual” innovation mindset and behavior. But this needs strong sponsorship, guidance and patience.

    DevOps must become the culture of the organization, driven from the top as well as the engineering teams, and people need to understand the implications for their role and responsibilities. CEOs and leaders need to actively support and drive the change, while providing the necessary working environments (tools and training) for those people who will be held accountable for operational stability. It is no good investing in technology and training for engineering to innovate if your ITSM functions are not brought along with the change, for example. But if you bring everyone with you, and address all concerns along the way, you will soon be reaping the rewards of the DevOps approach.

    About the Author / Romy Hughes

    Romy Hughes is director at Brightman. Romy is a business change specialist with expertise in service integration and management (SIAM) for managed services, business transformation and IT service management (ITSM and ITIL). Connect with her on LinkedIn.

  • SRE vs. DevOps — a False Distinction?

    SRE vs. DevOps — a False Distinction?

    Just a few days before he died at the beginning of the 1990s, a wise man taught us that “the show must go on.” Freddie Mercury’s parting words have long provided the guiding light for many, if not all, ops teams. In their eyes, the production environment should be exposed to minimum risk, even at the expense of new features and problem resolution.

    About 10 years ago, Google decided to change its approach to production management. It took the company only a few years to realize that while R&D focused on creating new features and pushing them to production, the Operations group was trying to keep production as stable as possible—the two teams were pulling in opposite directions. This tension arose due to the groups’ different backgrounds, skill sets, incentives and metrics by which they were measured.

    Trying to bridge this gap between the two groups, one of Google’s ops leaders, Ben Treynor, thought of an innovative solution. Instead of having an Ops team built solely from system administrators, software engineers—with an R&D background and mentality—could enrich the way the team worked with the development group, change its goals and help with automating solutions.

    Site Reliability Engineering, or SRE

    And so, the Site Reliability Engineering position was created. According to Google, SRE engineers are responsible for the stability of the production environment, but at the same time are committed to new features and operational improvement. Google decided its SRE teams should be composed of 50 percent software engineers and 50 percent system administrators. The engineers were driven to use software as a way of solving problems and perfecting what had historically been solved by hand. They integrated easily with the development team, and encouraged code quality improvements and automation testing.

    Understanding the value of SRE, several organizations of various sizes decided to embrace its principles. Some, such as Dropbox, Netflix and Github, are well-known for being at the forefront of technology leadership.

    Wait, Isn’t That DevOps?

    DevOps is a more recent movement, designed to help organizations’ IT department move in agile and performant ways. It builds a healthy working relationship between the Operations staff and Dev team, allowing each to see how their work influences and affects the other. By combining knowledge and effort, DevOps should produce a more robust, reliable, agile product.

    Both SRE and DevOps are methodologies addressing organizations’ needs for production operation management. But the differences between the two doctrines are quite significant: While DevOps raise problems and dispatch them to Dev to solve, the SRE approach is to find problems and solve some of them themselves. While DevOps teams would usually choose the more conservative approach, leaving the production environment untouched unless absolutely necessary, SREs are more confident in their ability to maintain a stable production environment and push for rapid changes and software updates. Not unlike the DevOps team, SREs also thrive on a stable production environment, but one of the SRE team’s goals is to improve performance and operational efficiency.

    Google tried a few approaches to implementing the SRE process before finding the one that suited them best. One of these approaches attempted to tie the number of permitted releases to the product’s stability. The principle underlying this process is that new releases are green-lighted based on current product performance.

    For each service, the SRE team sets a service-level agreement (SLA) that defines how reliable the system needs to be to end users. If the team agrees on a 99.9 percent SLA, that gives them an error budget of 0.1 percent. An error budget is exactly what the name suggests: the maximum allowable threshold for errors and outages. Here’s the interesting thing: the development team can “spend” this error budget in any way they like. If the product is currently running flawlessly, with few or no errors, they can launch whatever they want, whenever they want. Conversely, if they have met or exceeded the error budget, and are operating at or below the defined SLA, all new releases are frozen until they reduce the number of errors to a level that allows the launch to proceed. This process ensures that both the SREs and developers have a strong incentive to minimize the number of errors in production.

    Another interesting approach recommended by Treynor, which is more related to SRE professionalism and efficiency, is allowing SREs to move between projects. Moreover, he suggests allowing SRE engineers to move to development, and even the other way around. As the work done by both teams is similar, these transitions help the Ops team gain better and deeper knowledge of the product and code, and bring the Dev teams into the production space to help them understand its challenges. This strongly promotes a team atmosphere, rather than one in which an individual feels that “I’m on the SRE team for this product.” As part of this approach, Treynor had Dev teams handle 5 percent of the operations workload. This, according to many organizations, adds to the SRE team’s motivation and effectiveness.

    Riddles in the Dark

    Well, SRE seems perfect. As already stated, several large-scale organizations have chosen to move some of their production operations from old Ops to SRE. However, there are still a few questions that need to be asked:

    Is the position of SysAdmin (production/ops) no longer relevant for SysAdmins?

    Historically, almost all system administrators have come into their roles through tech support and similar work, or even just running Linux on their desktops and then transitioning into server work. It should be pretty clear that the same path is not available into SRE. To retain their positions, SysAdmins should now be more code-oriented, have better technological knowledge and be receptive to new methods of conducting the work they already do.

    Can an SRE team prevent production incidents?

    One of the strengths of a professional site reliability engineer is the ability to handle the growth of production load and traffic in-house. Monitoring and analyzing processes and logs with platforms such as the ELK Stack is part of the day-to-day workload, and the team should be able to identify problems as they occur, and even foresee risks to software stability. The power of this position—based on the skill set it requires—lies in developing solutions for these problems and risks. As Ciara, a software engineer in Google’s cloud storage SRE team, described it in a “Life in Google” post, “We solve cooler problems in cooler ways.”

    Are we doing DevOps or are we doing SRE?

    This question is usually asked by people trying to position themselves in the Operations world. According to many companies that implemented SRE in a slightly different way than Google, you don’t have to decide. At Reddit, ops engineers work on reducing toil, improving deployment and scaling processes, but they are referred to as “DevOps.”

    At Logz.io, we bridge the gap between developers and production through the use of automated monitoring and performance stress-testing. And we personally refer to it as “DevOps” rather than “SRE.”

    So, is it Just a Question of Context?

    In her blog, Charity Majors talks about SRE and DevOps being two different operational approaches that any organization can choose to work by, but insists on emphasizing that there is no “correct” approach. Although Google and Dropbox have decided on SRE, this does not mean the rest of the world should do so as well. What fits Google’s needs and organizational philosophy does not necessarily work for other orgs—at any scale. Moreover, Charity believes that DevOps, having grown and evolved within a broader variety of software organizations, is the more flexible, collaborative and adaptable approach, and will work better for most software organizations at all stages of development.

    On the other hand, SoundCloud senior engineer, Matthias Rampke issued a series of tweets on how SRE and DevOps were basically the same, with only one difference—high management support. Though in contradiction to her blog, Charity lent weight to this opinion during an AskMeAnything event that revolved around DevOps and SRE. She shared a story about a friend who was hiring for a startup, which, as an experiment, posted the exact same job description twice, the only difference being that one listing was titled “DevOps engineer,” and the other “SRE.” At the time of relating the story, DevOps was winning by 10 percent or 20 percent. However, all this means is that the job title is irrelevant, and naming an SRE team does not magically bestow qualities upon it. Rather than focusing on the title, organizations must focus on the work being done.

    To summarize, we can quote Matt Simmons, a technologist with many insights into this topic, who says, “Not every infrastructure needs an SRE, but every infrastructure could use an administrator who acted more like one.”

    — Asaf Yigal

  • SRE vs. DevOps vs. Cloud Native: The Server Cage Match

    SRE vs. DevOps vs. Cloud Native: The Server Cage Match

    This post supports my DevOpsDays Austin Presentation on May 4

    I don’t believe in DevOps shaming. Our community seems compelled to correct use of DevOps as an adjective for tools, teams and teapots. The frustration is reasonable: DevOps clearly taps into head space for both devs and operators who see a brighter automated future together. For example, check out this excellent DevOps discourse by Cindy Sridharan.

    As an industry, we crave artificial conflict so it’s natural to try and distill site reliability engineering (SRE), DevOps and cloud native into warring factions when they are not. They all share a focus on Lean process.

    Very simply: SRE is a job function, DevOps is a process approach and Cloud Native is an architecture.

    These three concepts have tremendous alignment because they all have the same fundamental goal: increase product delivery through improved flow. It’s really that simple. We are looking to enhance productivity using system thinking concepts from Lean manufacturing.

    SREs improve productivity by eliminating manual operations, reducing errors and improving infrastructure quality. They do this starting from the production infrastructure and working backwards. Highly automated delivery, integrated logs, systemwide monitoring and blameless root cause analysis are key tools. Before we get wrapped up in tools, the distinguishing factor for SRE teams is that they take ownership of the production infrastructure.

    There is tremendous power in assigning ownership for delivery. That’s why I see industry benefit in defining an SRE job function. A DevOps strategy is a system approach and it becomes an anti-pattern to making an individual or team responsible to own “the DevOps.”

    In DevOps, we look for ways to streamline the entire product creation flow. That often drives toward tools and techniques that SREs require; however, our goals are not department-specific. We want to work with all stakeholders including developers, product managers, testers, SREs, executives and customers. This system focus for developers becomes CI/CD pipelines, platforms, work tracking tools and other workflow elements. These tools are not specifically DevOps tools, but confusion is natural since they are often pulled into organizations due to DevOps thinking.

    This SREs should expect to collaborate deeply with other functions under a DevOps process.

    Cloud-native architecture also falls under the DevOps mandate because it strives to decompose large application stacks into highly automated collections of services. There is no magic “DevOps platform” in cloud-native design; however, teams using the architectural approach get modularity, strong APIs, service orientation, simplified configuration and build automation practices that fit perfectly into DevOps thinking.

    For example, using containers does not mean a dev or SRE team is following DevOps process or using cloud-native architecture; however, it is an indicator that they are leaning toward smaller deliverables and improved dev-production fidelity.

    Yet, SRE, Cloud Native and DevOps do have a shared enemy: Complexity.

    We’re doing a great job creating best practices and abstractions that accelerate developers and creating innovative infrastructure options in both cloud and physical. That means we are seeing dual explosions in developer productivity and infrastructure complexity. Unfortunately, this wealth has not made it into operations automation, which has been delivering incremental improvements at best. Even worse, the added workload and complexity adds pressure to SRE functions.

    Ultimately, all three of these concepts are aspects of Lean business practices; however, acceleration from DevOps and cloud native is outpacing progress by SREs. This imbalance creates risk for everyone involved because Lean is about system performance. Delays and failures cannot be isolated to a single silo.

    The mistake of seeing Dev and Ops as silos means creating false contention between SRE, DevOps and cloud native.

    If you are hearing that in your organization, then take it as a warning about how your teams are integrating Lean systems thinking into your processes.

     

    — Rob Hirschfeld

  • Is Site Reliability Engineering the True ‘Ops’ in DevOps?

    Is Site Reliability Engineering the True ‘Ops’ in DevOps?

    I recently had the privilege of attending the SRECon conference in San Francisco. It’s in its third year and 650 attendees gathered to learn and share their experiences, insights and site reliability engineering (SRE) practices.

    Most SRECon attendees either identified as SREs or were there to learn more about this new role and approach. While there was strong representation from the “unicorn” community (Google, Shopify, Twitter, Netflix and Linkedin), enterprises such as Capital One and Comcast were among the many exhibitors actively seeking to recruit SRE talent. The conference spirit was infused with enthusiasm, a willingness to share, innovative thinking and grassroots advice. It reminded me very much of my first DevOps Days in 2012. There is no doubt that interest in SRE is rapidly growing and worthy of attention by enterprise IT.

    The practice of site reliability engineering was born out of the book by the same name from Google. Interestingly, the SRE movement emerged separately from the DevOps movement—although there is little doubt that they are part of the same IT spectrum with similar customer value-driven goals. DevOps focuses on engineering continuous delivery to the point of  deployment; SRE focuses on engineering continuous operations at the point of customer consumption. Both domains rely on sharing, culture, metrics and automation. Both require human and automated resources to ensure a seamless value stream and exceptional customer experience.

    The role of the SRE is rapidly gaining momentum. Part systems administrator, part second tier support and part developer, SREs are encouraged to be inquisitive, to acquire new skills, to ask questions, to solve problems and to embrace automation. I asked several of the exhibitors what types of skills they were looking for in new SREs. The answers almost unilaterally included the ability to code as well as an unwavering thirst for knowledge and improvement—to find answers, solve problems and ensure the highest levels of reliability and availability.

    I also can see a clear bridge between SRE with IT service management (ITSM). Many of the presenters spoke about the handling and management of late-night incidents, escalations, root cause analysis, service level objectives, availability metrics and other topics addressed in ITSM frameworks such as ITIL. The tools and process definitions may be different, the opportunity to instill agility into many of the ITSM processes to meet changing demands is certainly apparent.

    What are the risks? To my mind the biggest risk is if SRE, DevOps, ITSM, Agile SDLC and other frameworks do not align and remain siloed outposts. SRE and DevOps are two sides of the same coin. Let’s hope that the SRE momentum catches up to DevOps and that enterprises begin to integrate both sets of practices as the norm.

    — Jayne Groll

  • Strategies to Build Trust in DevOps

    Strategies to Build Trust in DevOps

    I’ve been a member of the DevOps community a long time (literally since before it was ever even called “DevOps”). I was a pre-publication reviewer for both “The Phoenix Project” and the recently published “The DevOps Handbook,” and I’m always interested in seeing the outcomes successes software teams have and the failures they overcome, including trust.

    While DevOps has proven to be effective in organizations of all sizes—startups and enterprises alike—there are still many who have not yet begun their DevOps journey, so I thought I might share some of my own experiences while introducing DevOps into an enterprise IT group.

    When I was the VP of Operations at Big Fish Games, I had the responsibility for production operations at Big Fish’s three production data centers in Seattle, Virginia and Luxembourg. I also had responsibility for internal corporate IT and information security. Our production operations supported around a million game downloads per day. Games were localized into 11 different languages. We processed essentially every major currency, and right before I left Big Fish, we started taking Bitcoin as well. From an IT perspective, Big Fish Games was a global eCommerce company selling digital widgets on the internet. We had around 800 employees in our five offices in Seattle, Ireland, Oakland, Luxembourg and Vancouver, BC.

    My first attempt to introduce DevOps at Big Fish failed miserably due to cultural misunderstandings and a fundamental lack of trust between the development and operations teams.

    You absolutely need some minimum baseline level of trust in your organization if you are to have any chance at implementing DevOps. Trust is important because you are blurring traditional boundaries between development and operations responsibilities. Including development in production operations can be scary for operations teams. It just feels wrong from a historical ITIL-type perspective, and operations teams may know development as “the people who always break things.” At the same time, it can be scary for development to involve operations in the development process because they may know operations as “the team that always says ‘No.’”

    Building Trust

    Successfully merging the priorities and goals of development and operations teams to create one cohesive DevOps effort can fail due to cultural misunderstandings and, oftentimes, a fundamental lack of trust. I used three specific strategies to overcome these challenges.

    Lose the Term ‘DevOps’

    At Big Fish Games, application development teams equated DevOps with developers carrying pagers: an idea that they really did not like. They were highly resistant to this. Every time I brought up DevOps, they thought I was engaged in a secret plot to make the developers carry pagers. What I found interesting about this was that I had no intention of asking developers to carry pagers.

    The term “DevOps” covers a large range of continuous improvement techniques. Some organizations have developers carry pagers, and it works for them, but this is certainly not a prerequisite for DevOps. What was worse, I had no idea they were thinking this because they were afraid to bring it up for discussion. We failed in our first attempts to implement DevOps because the trust level between development and operations teams was so low that we couldn’t even tell each other what we were thinking.

    I realized two things from this experience:

    • Getting rid of a term that scared both development and operations teams would eliminate an unnecessary distraction preventing these teams from coming together to focus on common goals.
    • We needed to do something about the trust issues between development and operations before we could succeed with DevOps.

    Once I realized this, I immediately stopped using the word “DevOps.” Instead, I talked about continuous improvement and specific continuous improvement methodologies that we wanted to implement, and how these new adoptions would result in shared, organizational-wide growth.

    Focus on Methods That Benefit Other Teams

    When deciding which of these DevOps, a.k.a. “continuous improvement,” techniques we wanted to implement, I specifically chose ones that would benefit development teams as opposed to ones benefiting operations or other teams. The development teams noticed this over time. They could see I was doing this and that I (the person who ran operations) was going out of my way to help them. This helped build trust. This wasn’t a one-time thing. We did this consistently.

    Here’s another simple failure example: I noticed while managing release efforts that release schedules for the various development teams ranged from a couple weeks to a couple months—too long for a hyper-competitive industry like gaming. I asked the development teams if they would like our release engineering team to support daily code releases or even multiple code releases per day. I thought this was an innocuous question. In fact, I thought I was being a little clever by playing dumb here. I assumed I would get an enthusiastic high-five along with a, “Yes, we’d love that! How soon can you make that happen?!”

    I was surprised when I received a violent “No! We don’t want that!” This confused me for a while until I realized that the development team thought I was in cahoots with the product management team to put the squeeze on them. In my mind, I was offering to make their release process more lightweight so they could break up their work into smaller, more frequent releases with less stress for everyone, but I didn’t understand their fears and problems. I wasn’t looking at it from their perspective and here was the trust issue rearing up again. They misinterpreted my intentions as nefarious.

    I pictured them being able to release code whenever they want, which seemed like something they would like. In their minds, they pictured a product manager constantly randomizing them with demands for new features to be released right there while the product manager was leaning over their shoulder. This was not the kind of problem you could resolve with logic. The solutions and working procedures I was proposing were so far outside the development team’s experience that they couldn’t extrapolate through simple logic from their current state to the world I was describing.

    Maximize Daily Interactions Between Development and Operations Teams

    It’s easy to distrust someone you don’t know personally or you don’t have to see every day. It’s a lot more difficult to hate on someone you know and work with on a daily basis.

    One of the DevOps techniques that I implemented early on was embedding a dedicated operations liaison into each development team. This liaison acted as an advocate for that development team back into operations and they helped smaller development teams navigate their work through a large, complex, centralized operations organization. The operations liaison attended daily development team standups, group meetings and project meetings. This program was a huge success. Development teams saw how their operations liaison helped expedite their work through the operations organization and advocated for their development team’s interests inside operations.

    After the development teams got used to working with their operations liaison on a daily basis, we pushed a bit farther and got the development teams to agree to include operations in architectural discussions at the beginning of application design. At first, our development teams were skeptical that operations could provide useful input here, but the important thing is we had developed enough mutual trust that they were not fearful of including operations in the discussions. They were skeptical that we would add any value, but they weren’t afraid that we would cause them problems. This was a big win.

    Although it felt a little weird to our development teams to include operations in design discussions, they quickly saw how obtaining input from operations early on, before a product was built, resulted in better quality and fewer headaches for everyone when we launched into production. I also made sure that we put our most senior and diplomatic operations team members into this role.

    In conclusion, it is worth repeating that trust is a prerequisite for any organization to succeed with DevOps. This is due not only to problems arising from fuzzy definitions of what DevOps is and is not, but also the seemingly dangerous idea of blurring the boundaries between development and operations organizations. Whatever your definition of DevOps is, fostering an increased level of trust between development and operations teams as a prerequisite for your DevOps journey will ensure that even greater trust—and the positive business impact that comes with it—are your outcomes as well.

    Good luck!

    About the Author / Paul Farrall

    Paul Farrall is the VP of Operations for Skytap. He was most recently vice president of Operations at Big Fish Games where he led the rollout of global infrastructure for the Big Fish cloud gaming platform and implementation of the Big Fish DevOps program. Prior to Big Fish, Paul was the Director of Operations and Chief Information Security Officer at Intelius, a Bellevue e-commerce company. Previous companies also include AT&T Wireless in Bothell, Washington, and GetThere.com in Palo Alto, California (now Sabre). Connect with him on LinkedIn and Twitter.

     

  • Automation: Achieving Faster Rollouts with Better Accuracy

    Automation: Achieving Faster Rollouts with Better Accuracy

    A changing landscape requires new processes

    At the recent Symposium/ITxpo, Gartner shared a list of technologies and trends that will be important for organizations to embrace and prepare for in 2017. These trends, which are listed in the report, “Top Strategic Predictions for 2017 and Beyond: Surviving the Storm Winds of Digital Disruption,” reinforce the breadth of technology that IT must support, roll out and engage with. This includes everything from the 100 million consumers who will shop using augmented reality to dealing with the roughly 21 billion endpoints expected to be in use in 2020.

    The report also demonstrates how fast technology changes directions. For example, Gartner predicts that in 2019, 20 percent of brands are expected to abandon their mobile apps. “Many brands are finding that the level of adoption, customer engagement and return on investment (ROI) delivered by their mobile applications are significantly less than the expectations that underpinned their app investment,” the report notes. However, this doesn’t mean less work. IT departments continuously have shift gears to support new processes, news apps and new technologies, such as the Internet of Things (IoT) and the massive amount of data that it produces.

    The dynamic needs of today’s businesses are driving organizations to increasingly adopt new technologies and applications that oftentimes don’t play nice with the existing set of infrastructure and legacy systems in place. So, while it’s clear that the technology landscape is changing, what isn’t so obvious is how IT and business decision makers should be adapting to an industry that is in constant motion.

    In-Steps DevOps

    To keep up, IT must implement agile methodologies that can adapt quickly to market changes. The DevOps mentality is making its way into many organizations and involves the development team creating, building and evolving products to ensure the organization is up to date, while the operations team confirms everything is running smoothly prior to a launch going live. This level of checks and balances isn’t just a nice to have, it’s a necessity—one mistake and the entire system can be brought down. Fostering a greater level of collaboration between development and operations makes it possible to efficiently and effectively build, test and correct new features and new products.

    While the shift to a more collaborative environment between the Dev and Ops teams can take time, using modern IT automation solutions can make this transition easier. Such solutions automate many of the routine—and even not so routine—elements of the application release process within DevOps. Many of the actions necessary to move from code to production—not to mention maintenance, testing and cleanup tasks—can be taken off the backs of developers and operations, saving them time so they can focus on higher-level projects and reduce manual errors.

    Finding the Balance Between Speed and Accuracy

    In 2012, Facebook announced it was increasing the speed at which it released code to twice a day, every day. As Chuck Rossi, director of Release Engineering at Facebook said, his team follows the advice “ship early and ship often.” Facebook continues to follow this model, and while the company certainly didn’t invent it, the model has caught on and is spreading among companies developing virtually every type of technology. The catch? It’s an incredibly time-intensive process and not every company has the resources necessary for a dedicated release team. Plus, a true shift to DevOps doesn’t happen overnight, especially since there is a ramp-up time for folks to get on board. For example, Phil Dibowitz, systems engineer at Facebook, told TechTarget that it “led its Chef team through a three-year process to rethink configuration management to scale on par with the web-scale company’s data centers.”  This took three years, he said, “not because of the technological change, but because of the structural change.”

    A slow transition to a new technology or new way of operating is a common occurrence because historically the production systems are cordoned off from development groups. While the Ops team has to ensure the rollout will meet certain standards, the Dev team wants to speed the process to begin the final rollout process as soon as possible. A happy medium exists, but if it isn’t found, things can go very wrong. Unfortunately, 2015 gave us two major examples of what can go wrong if products can’t speak to each other and rollouts are completed too early: Remember the New York Stock Exchange and United debacles?

    Breaking Constraints Through Automation

    This is where modern workload automation comes into play. Automation and solid change management allows IT to flexibly add on new technologies with automated processes for building, testing, upgrading and configuring IT—all while reducing the slack time between various steps. For instance, a workload automation solution should be able to trigger a build, deploy the build to the test environment, run the test suite and, assuming testing meets acceptable criteria, promote the build to production. If something does go wrong, necessary remediation can occur, and through monitoring and alerting capabilities, the required parties can be notified proactively.

    Additionally, automation solutions with pre-built integrations make it easier for disparate applications to communicate, creating an end-to-end approach to workflow creation and maintenance. As a result, IT organizations can minimize the risk of business disruptions and achieve a more fluid technology delivery to the business.

    Although it’s not a silver bullet that will solve all DevOps transition issues, automation does reduce manual intervention, which leads to fewer delays and errors and improves speed at which tasks are completed. However, beyond the technological aspects, there must be a clear understanding by both sides (the Dev and Ops group) on what their counterparts need—and this requires a high level of trust and coordination.

    While the specifics around the DevOps methodology certainly will change, automation will remain a critical piece of the process. With IT automaton, DevOps teams are able to refocus on critical IT business and operational processes, thereby reducing delays, decreasing errors and improving the speed at which tasks are completed. Ultimately, automation makes it possible to increase the speed of product releases and enable companies to stay ahead in the age of unpredictable customer preferences and ever-changing technological developments.

    About the Author / Mehul Amin

    Mehul Amin, a 21-year veteran of Advanced Systems Concepts Inc. (ASCI), previously held roles as product engineer and technical support manager and played an integral part in building the company’s workload automation solution, ActiveBatch. In his current role as director of Engineering, Mehul drives high-profile initiatives and leads both the Product Engineering and Release Engineering teams. Mehul received a master’s in computer science from the Stevens Institute of Technology.

  • The Code Doesn’t Lie, and Other Operations Mantras

    The Code Doesn’t Lie, and Other Operations Mantras

    As engineers, we spend a lot of time talking about things such as release processes, QA environments and deployments. But at the end of the day, most software systems fail because the software itself is faulty. The code doesn’t lie—you can almost always find the solution to your problem within the code itself. Of course, looking to the code for answers is a time-consuming process if you’ve deployed too much in one batch. After all, sometimes the tiniest mistake can derail a whole string of code, and having to sort through a haystack of code looking for a tiny needle is not fun.

    There are a few corollaries to this mantra that “the code doesn’t lie”:

    • “Know your data like you know yourself.” Sometimes the quality of the data itself can cause a system to behave differently.
    • “A feature is a bug in a tuxedo,” which is largely for you product folks out there.
    • “Where there is one bug, there are many.” When you find a problem and it leads to a systemic set of problems, it may seem terrible at first, but the reality is, you have the opportunity to fix a lot in a short amount of time.

    I had a chemistry teacher who used to say, “A mole is a mole is a mole,” referring to Avogadro’s number. My version is: “A bug is a bug is a bug.” A feature that behaves incorrectly (even if built to spec) is a bug. A badly performing system is a bug. A transient failure is a bug. A race condition is a bug. No matter what they look like, it’s our job to squash the bugs.

    Branch Readiness is a Joke

    A very talented engineering director once made the comment, “Branch readiness is a joke,” about the branch/development model. As 23 different teams were preparing their branches for integration onto the main trunk, code changes were flying around with very little testing/certification. We even had code that would not compile.

    When all 23 branches hit the main release branch, all hell broke loose. And this is where the actual functional, integration and performance testing began. The key problem with this model is that integration and certification happened late in the game. The other problem was that we did not abide by our own criteria of what constituted branch readiness; instead, we were constantly softening the restrictions until bad code made its way into the system.

    We moved to a trunk-based model of development where check-ins are made directly to the trunk, and all code must compile and be pretested. Continuous integration and testing are run against the trunk all day and all night. We can ship from the trunk at any moment.

    Now you may be asking yourself, What does this have to do with operations? The answer is: everything. We can control what changes get to the site. We can certify what changes get to the site. We are assured at every moment of a minimum quality in the changes that do get to the site. These smaller changes give us the ability to move bits onto the site more frequently. We are not forced into large-scale, monolithic deployments, which inherently create more risk to the site itself, often with little ability to roll back.

    Learning from ‘Branch Readiness is a Joke’

    Just because the code doesn’t lie does not mean that you can always understand all of its implications. The key is limiting the amount of code being changed in any individual release to an amount an engineer or automated system can understand. By avoiding large, monolithic releases containing thousands of changes and instead testing, verifying and considering each change individually, we can understand what the code is telling us. We can release to the trunk with confidence that what we build and deploy is going to work.

    The One-Character Change

    We live in a world of bits and bytes (eight bits per byte). One byte is the equivalent of one character. In some cases, a one-character change is benign. For example, changing the number of members of a site on an informational page from 134M to 135M (4 to 5) will not cause any harm to the site.

    But sometimes, a one-character change can be disastrous. Consider a DNS change that has a one-character mistake for www.yourcompany.com. Get it wrong and you are off the air. It is very important to understand the impact of a change going awry. If it can cause a large impact, then we need to ensure we understand the change and have a clear plan to roll it back if we need to.

    Here’s a great example of a very small change going sideways very fast: We once scheduled a maintenance to test routing traffic through a European point-of-presence. Unfortunately, the time-to-live parameter was set to a large number (hours instead of minutes) on the DNS entry. The result was that the change, once implemented, couldn’t be fully undone for hours. It was a one-character change with a bad outcome.

    Learning from the One-Character Change

    The code doesn’t lie, but that only matters if we are paying attention. Put simply, not all changes are equal. By taking the time to think about what a change is intended to accomplish and the various ways it could go wrong, we will always be better off than if we didn’t do due diligence. The one-character change can be benign, or it can be catastrophic. It’s important to figure out the possible ramifications of any changes (no matter how small they may seem) before proceeding.

    Operations tends to be the first group to get called after hours when something goes wrong. As a result, we have a vested interest in quality of code shipping to our site. By working to improve the code quality before it hits production, you can significantly reduce the number of problems you encounter.

    Start with where the code gets committed into source control. For each change made ask the simple questions: Does the code still compile? Does your application still build correctly? Does it still work as intended? By automating the process of asking these questions for every change, we gain the ability to deploy new versions of our application based on any commit.

    Once we have good code quality making it into source control, we can begin making further improvements through the use of a canary process and start focusing on things such as performance. Remember, the code doesn’t lie. If we can understand each change, then we can start preventing problems before they begin.

    This post is part of the series “Every Day Is Monday in Operations.” Throughout this series we discuss our challenges, share our war stories and walk through the learning we’ve gained as Operations leaders. You can read the introduction and find links to the rest of the series here.

    About the Author / David Henke

    David Henke has more than 35 years of experience working in technology, including senior engineering leadership positions at LinkedIn, Yahoo!, and AltaVista Company. He’s also been a founder at two different software companies, both of which were acquired. Currently, David serves in a variety of board and advisory positions with organizations like NerdWallet and UC Santa Barbara.

  • An Operational Approach to Benchmarking Microservices

    An Operational Approach to Benchmarking Microservices

    Benchmarking microservices often falls behind on the engineering team’s to-do list. However, those that aren’t are missing out on a key component. The fundamental goal of benchmarking is to better understand the software, and test out the effects of various optimization techniques for microservices. In this post, I’ll describe an effective approach to benchmarking microservices.

    First, create a spreadsheet for tracking your benchmarking. One convenient way to document a series of benchmarks is in a Google Spreadsheet. It allows collaboration and provides the necessary features to analyze and sum up your results. Structure your spreadsheet as follows:

    • Title page
      • Goals
      • Methodology
      • List of planned and completed experiments (evolving as you learn more)
      • Insights
    • Additional pages
      • Detailed benchmark results for various experiments

    Then, before you engage in benchmarking, clearly state (and document) your goal. Examples of goals are:

    • “I am trying to understand how input X affects metric Y”
    • “I am running experiments A, B and C to increase/decrease metric X”

    Next, pick one key metric, stating clearly which one metric you are concerned about and how the metric affects users of the system. If you choose to capture additional metrics for your test runs, ensure that the key metrics stands out.

    Now, you’re going to have to think like a scientist and perform a series of experiments to better understand which inputs affect your key metric and how. Consider and document the variables you devise and create a standard control set to compare against. Design your series of experiments in a fashion that leads to the understanding in the least amount of time and effort. You’ll then need to define a methodology for running your benchmarks. It is critical your benchmarks be:

    • Fairly fast (several minutes, ideally)
    • Reproducible in the exact same manner, even months later
    • Documented well enough so another person can repeat them and get identical results

    Document your methodology in detail. Also document how to re-create your environment. Include all details another person needs to know:

    • Versions used
    • Feature flags and other configuration
    • Instance types and any other environmental details

    In most cases, to accomplish repeatable, rapid-fire experiments, you need a synthetic load generation tool. Find out whether one already exists. If not, you may need to write one. Understand that load generation tools are at best an approximation of what is going on in production. The better the approximation, the more relevant the results you’re going to obtain. If you find yourself drawing insights from benchmarks that do not translate into production, revisit your load generation tool.

    It is then important to validate your benchmarking methodology, and repeat a baseline benchmark at least 10 times and calculate the standard deviation over the results. You can use the following spreadsheet formula:

    =STDEV(<range>)/AVERAGE(<range>)

    Format this number as a percentage, and you’ll see how big the relative variance in your result set is. Ideally, you want this value to be less than 10 percent. If your benchmarks have larger variance, revisit your methodology. You may need to tweak factors like:

    • Increase the duration of the tests.
    • Eliminate variance from the environments.
      • Ensure all benchmarks start in the same state (i.e. cold caches, freshly launched JVMs, etc).
      • Consider the effects of Hotspot/JITs.
    • Simplify/stub components and dependencies on other microservices that add variance but aren’t key to your benchmark.
      • Don’t be shy to make hacky code changes and push binaries you’d never ship to production.

    Important: Determine the number of results you need to get the standard deviation below a good threshold. Run each of your actual benchmarks at least that many times. Otherwise, your results may be too random.

    Now that you have developed a sound methodology, it’s time to gather data. Some tips include:

    • Only vary one input/knob/configuration setting at a time.
    • For every run of the benchmark, capture start and end time. This will help you correlate it to logs and metrics later.
    • If you’re unsure whether the input will actually affect your metric, try extreme values to confirm it’s worth running a series.
    • Script the execution of the benchmarks and collection of metrics.
    • Interleave your benchmarks to make sure what you’re observing aren’t slow changes in your test environment. Instead of running AAAABBBBCCCC, run ABCABCABCABC.

    Now, you’ll need to create enough loads to be able to measure a difference. There are two different strategies for generating load:

    1. Redline it! In most cases, you want to ensure you’re creating enough load to saturate your component. If you do not manage to accomplish that, how would you see that you increased its throughput? If your component falls apart at redline (i.e. OOMs, throughput drops, or otherwise spirals out of control), understand why, and fix the problem.
    2. Measure machine resources. In cases where you cannot redline the component, or you have reason to believe it behaves substantially different in less-than-100-percent-load situations, you may need to resort to OS metrics such as CPU utilization and IOPS to determine whether you’ve made a change. Make sure your load is large enough for changes to be visible. If your load causes 3 percent CPU utilization, a 50 percent improvement in performance will be lost in the noise. Try different amounts of load and find a sweet spot, where your OS metric measurement is sensitive enough.

    As you execute your benchmarks and develop a better understanding of the system, you are likely to discover new factors that may impact your key metric. Add new experiments to your list and prioritize them over the previous ones if needed.

    In some instances, the code may not have configuration or control knobs for the inputs you want to vary. Find the fastest way to change the input, even if it means hacking the code, commenting out sections or otherwise manipulating the code in ways that wouldn’t be “kosher” for merges into master. Remember: The goal here is to get answers as quickly as possible, not to write production-quality code—that comes later, once we have our answers.

    Once you’ve completed a series of benchmarks, take a step back and think about what the data is telling you about the system you’re benchmarking. Document your insights and how the data backs them up. It may be helpful to:

    • Calculate the average for each series of benchmarks you ran and to use that to calculate the difference (in percent) between series — i.e. “when I doubled the number of threads, QPS increased by 23 percent on average.”
    • Graph your results — is the relationship between your input and the performance metric linear? Logarithmic? Bell curve?

    Finally, you’ll need to present your insights. Here are some final tips for exhibiting to management and/or other engineering teams:

    • Apply the Pyramid Principle. Engineers often make the mistake of explaining methodology, results and concluding with the insights. It is preferable to reverse the order and start with the insight. Then, if needed/requested, explain methodology and how the data supports your insight.
    • Omit nitty-gritty details of any experiments that didn’t lead to interesting insights.
    • Avoid jargon, and if you cannot, explain it. Don’t assume your audience knows the jargon.
    • Make sure your graphs have meaningful, human-readable units.
    • Make sure your graphs can be read when projected onto a screen or TV.

    — Stefan Zier

  • Making Performance More Than a Best Practice

    Making Performance More Than a Best Practice

    Yes, you read that correctly: SRE means that strong application performance—speed, reliability and availability—no longer is simply a best practice. It is a must. Site reliability engineering, or SRE, is a term Google coined several years ago, and while there is no universal definition, SRE can be thought of as a discipline that blends software engineering with IT operations management. At the heart of the SRE model is a site reliability engineer who knows how to code as well as handle ops. The SRE concept is similar to DevOps, which is intended to bridge the divide between development and operations teams, but until now has not caught on the same way DevOps has.

    The Need for SRE

    The rise of the digital economy is now propelling end-user application performance to the top of the IT priority list. According to a recent EMA Survey, 55 percent of IT professionals rank end-user experience monitoring as their top application management priority. At the same time, growing IT complexity, including the cloud and multi-platform applications, are making it more difficult than ever to ensure strong user performance and identify the root source of problems before they escalate. We believe this performance imperative is driving the SRE resurgence, giving DevOps teams the necessary “teeth” to govern and resolve performance-related disconnects, swiftly and decisively.

    Performance issues traditionally have been a lightning rod of conflict for even the most well-oiled DevOps teams. Developers, relentlessly focused on rolling out new products and features, keep throwing their “completed” work over the fence to operations. Operations teams, conversely, are focused on minimizing change and maintaining stability as much as possible. These opposing desires naturally create friction.

    Performance problems only exacerbate this. Developers, who have already moved on to the next must-have feature, say it’s operations’ fault. Operations points their fingers at the developers, saying that poorly written code is to blame. No one is assuming responsibility and the developers and operations teams are stuck in a stalemate. Meanwhile, users are suffering, the call center is being inundated, revenues likely are circling the proverbial drain and the situation is quickly reaching a boiling point.

    SRE is valuable because it finally brings an end to this suffering through a set of clear, irrefutable, mathematically based rules to govern all DevOps efforts and decisions. If a new piece of software achieves a targeted performance level, further rollouts can proceed. If not, future releases will halt until the first piece of software is fixed. The site reliability engineer, who is well-versed in both development and operations, serves as an independent arbiter who provides a clear assessment of where the root cause lies—on the development or operations side of the house. With guidelines firmly established and a central, unbiased person in charge of determining the right course of action, the developer/operations friction can be avoided and problem resolution can commence.

    In an SRE model, developers and IT operations also share a staffing pool. If there’s budget for one hire, and a product is not delivering an agreed-upon performance level, the hire goes to the IT operations. If the product is delivering the required performance level, the hire can go to development, which will help the development team roll out new features and products.

    The real beauty of SRE is that both sides have a vested interest in achieving strong application performance, ideally right out of the gate. If whichever side owns the problem does not fix it, they can forfeit any hopes of adding manpower to their team, at least in the near-term. There’s another more serious implication: for developers, there will be no further development; and for operations teams, less software to support may translate to lower headcount requirements.

    An SRE approach has many advantages, including fostering a vested interest in performance across the entire DevOps team, and holding feet to the fire. Performance cannot be an afterthought—performance must be considered and measured from the very beginning phases of ideation and development all the way through production and post-production modifications. Users don’t care how fast or efficient your DevOps team was if the ultimate work product does not perform—in fact, this means the whole effort likely was in vain. Like a strict adjudicator, SRE may just be what DevOps teams need to bring to an end to performance-related squabbles once and for all.

    About the Author / Mehdi Daoudi

    mehdidaoudiMehdi Daoudi is CEO and co-founder of Catchpoint Systems, provider of web performance testing and monitoring solutions. Before Catchpoint Systems, he spent 10+ years at DoubleClick and Google, where he was responsible for Quality of Services, buying, building, deploying, and using monitoring solutions to keep an eye on an infrastructure that delivered billions of transactions daily. Connect with him on LinkedIn and Twitter.

  • Keep Applications Running by Testing Early, Continuously

    Keep Applications Running by Testing Early, Continuously

    Almost every product we buy or use today goes through some form of extensive testing before it’s released on the market; typically, these tests are designed to replicate the worst-case scenarios that a product may encounter under real-world conditions.

    Automobiles, for example, go through extensive hot and cold weather testing in remote locations, and are driven over obstacle-laden tracks designed to push the limits of their durability. Even seemingly mundane objects such as children’s toys are exposed to abusive simulations to ensure that they don’t harbor any potentially hidden and harmful effects.

    On the current landscape, applications are the face of your business, often serving as the battleground where consumer loyalty is won or lost. Given this reality, it seems crazy to think that any applications are released having only been exposed to a minimum of tests. Yet, in the name of getting to market quickly, that’s far too often the case.

    As disciplines including agile and continuous delivery continue to increase the speed at which development teams can release software, it’s easy to put the crunch on testing in the rush to get an update released. But that’s a costly mistake in the long run, as research has shown that it is 50 percent less expensive to find an error in testing than to address the same error once it has reached production, and more than 90 percent cheaper  when addressed in the development/coding phase.

    Given the dramatic cost savings of addressing errors in development, clearly the goal of any organization focused on continuous delivery should be to shift testing left so a greater number of defects are found early in the development process. This is what we refer to as continuous testing, and it’s a huge part of the reason why I am so excited about CA’s acquisition of BlazeMeter.

    Performance load testing should be a core element of any continuous testing strategy because it is a critical factor in determining system reliability. Similar to the worst-case scenario testing that consumer products undergo, performance testing can quickly uncover applications defects that might otherwise go unnoticed. After all, the manner in which a system operates with a few hundred concurrent users may be very different than how it runs with thousands of users.

    BlazeMeter enables today’s development, operations and DevOps teams to conduct this critical performance testing earlier in the development process, saving countless hours and resources. The company is also competing—and winning—in a highly competitive marketplace where it has been pitted against much larger, more entrenched competitors.

    Part of the reason BlazeMeter is winning is its focus on offering a SaaS-based solution that is highly cost-effective and easy to engage. Using this model, customers can try  out the testing software with a few clicks, and then quickly move forward to broadly adopt the system. Another catalyst to its success: BlazeMeter was built using open source tools, such as Apache JMeter, used widely in the development community for performance and load testing. As a result, BlazeMeter can easily run and analyze JMeter and other open source-based load tests—such as Selenium, Gatling and Locust—within a commercial-grade performance testing system.

    This combination of a SaaS delivery model and incorporation of open-source tools, augmenting strong performance testing functionality, make BlazeMeter very complementary to CA’s existing continuous delivery solutions, including its CA Service Virtualization, CA Test Data Manager, CA Agile Requirements Designer and CA Release Automation products.

    At CA, we’re committed to providing the most comprehensive, integrated and open continuous delivery ecosystem, allowing our customers to release swiftly, test agilely and constantly increase reliably. The addition of BlazeMeter will further enable customers test continuously, thereby increasing agility and the overall quality of their software, to ensure that applications don’t break down on the side of the virtual road.

    — Aruna Ravichandran