devopsy.kinsta.cloud

Author: Patrick Hubbard

  • Did Your Last DevOps Strategy Fail? Try Again

    Did Your Last DevOps Strategy Fail? Try Again

    Let’s get one thing out of the way. DevOps isn’t a panacea—it isn’t magic, it isn’t a requirement for any enterprise and it won’t work for every team in every situation. Above all, it isn’t a yardstick by which an outsider can measure great operations, development or IT. And if anybody tries to tell you it is, feel free to say thanks and politely change the subject. You’re the one responsible for delivering the services your business depends on, and if the way you do it meets the needs of your business and the team is happy, stick with it.

    But if you’ve tried adopting DevOps methodology, it’s likely because your business needs to substantially alter itself to meet specific goals. And DevOps has demonstrated the ability to help many organizations. Unfortunately, just as many seem to abandon DevOps initiatives and roll back, sometimes more frustrated than if they hadn’t tried it in the first place.

    IT might be the team responsible for reliability, but this is a big challenge in an increasingly “fail fast” era. Many businesses are encouraging their departments to experiment, learn from their mistakes and keep trying until they deliver business-transforming innovation. But in IT, the high visibility of issues—even those initially within the accepted risks of a new project—seem to be treated differently. Even minor and expected teething pains can be judged as failures in hindsight. And many of us have seen instances in which attempting something new and encountering difficulty seems to make it five times harder to get off the ground on the next attempt.

    DevOps is a prime example. As its culture and practices gained popularity, companies were quick to jump on the bandwagon—only to stumble in the execution or not deliver anticipated “transformation” or measurable ROI. It’s not uncommon for these companies and decision-makers to be left with a bad taste in their mouths. They may be reluctant to revisit DevOps and try again, even with the gained value of previous investments and lessons learned.

    This isn’t much of a surprise: Humans don’t like to investigate public failures, and everything in IT is public, assessed in binary absolutes. It either works or it doesn’t, apps are up or down, and everyone in the company can see them. It’s also natural for businesses to switch their focus to the next potential success if personal political capital was part of a DevOps project. And as engineers, looking forward is in our DNA. We’ve built our careers around finding new and better ways to meet customers’ needs, solving novel challenges, and permanently squashing nagging issues to make room for new technology. The trick is to avoid considering DevOps a passing fad.

    There’s significant value to be gleaned from any investment in DevOps—a better understanding of the human dynamics of your organization, at a minimum. If your team tried to implement some of the processes of DevOps or took the advanced step of adopting DevOps culture, don’t forget you did it for a reason. Go back and review what led you to consider DevOps in the first place. Chances are the business needs behind it haven’t changed and may have even expanded.

    Outline those earlier reasons—they’re likely still there, even under novel circumstances such as shifting the team to work from home. Did your DevOps effort meet its original objectives, regardless of any rear-view assessment? Drill into the details where it didn’t. Was it due to culture, communication, risk reassessment or operations challenges? Did your team end up building something different than what you intended—a compromise not meeting the business’s needs? Take your time; you likely had many good people on the effort who are eager to share what they learned. A previous misstep could be due to a skills gap, software, management support or something as basic as infrastructure. The first step in taking another swing at DevOps is analyzing the previous attempt with data instead of opinions.

    And there’s a good reason you’re likely planning to try again: The stakes are high. According to Forrester, 80% of revenue growth will depend on digital offerings and operations by 2022. Organizations will survive and grow depending on their ability to design and deliver technology services, respond to issues and meet growing user expectations. At the same time, considering the environment brought on by COVID-19, IT teams are seeking new ways to sustain operations and use the time to build a head start for when business picks up.

    With this in mind, here are a few ways businesses can approach evaluating the execution of a first DevOps attempt to inform a successful second try.

    First: Deputize an Investigator

    You’ll need a deputy to help you conduct a forensic analysis and your QA team might be an unexpected place to find the right candidate. QA likes many of the outcomes of DevOps practices because they’re often suffering horror-show weekends at the hands of big-bang monolithic deployments at several companies. If you don’t find a detective there, empower someone who focuses on quality service and considers it a peer to availability, security and resiliency.

    Your first job will be to demark the fault lines. Remember, the task isn’t to assign blame and a good admin-side manner is required to prevent this. Remind the team the mission is to avoid making the same mistakes, not to point fingers. Perhaps you originally had fantastic buy-in but everyone got caught up in the wrong details. Was the problem operational? You’ll need to know.

    Second: Admit Mistakes and Ask for Feedback

    Once you’ve conducted your analysis, you’re ready to take it to the next step: humanizing past technical actions. Consider what the previous attempt might have been like for different teams and people. Remember what it was like when you rode the help desk, when you were sitting in the data center on a weekend or when you were working as a dev lead under an impossible deadline. Chances are you’ll remember the common frustration was external direction not taking people into account. DevOps success is primarily about people. Align your DevOps reboot with the humans on your team and the processes will follow.

    For instance, your company may have hired outside consultants to “rub a little DevOps on it,” and it’s possible they evangelized a lot of dogma. “Do exactly these things, this exact way, or you’ll fail at digital transformation,” they might have said. There may have even been FUD and frightening charts. Perhaps they said everything would be neatly wrapped up in six months and you’d achieve a high-performing, blended, and fully active nirvana within a year. “You could be the next Netflix!” You’ve probably heard this story several times by now, and if it’s happened to you, know you’re not alone.

    But your approach this time will be different because it will be yours. You will have noted sharp edges needing to be filed down. The broader team will be integral to conversations from the get-go. Internal champions won’t be under the influence of ballyhoo, fancy decks or hard sells. Your team will have a list of concerns but also possibilities reflecting a shared interest in modernizing operations. One look at salaries for job postings with “DevOps” in the skill set has likely piqued more interest from your team than a year ago.

    Open conversation has always been the hallmark of good forensic analysis. Express what you think went wrong the last time and encourage frankness. Acknowledge concerns—previous outcomes may have validated many of them. Your team will be looking for proof you’ve given thought to a process that might—in extreme circumstances—temporarily affect availability or lead to a security concern. Maintain an open invitation for everyone on the team to be a part of the conversation and hear their concerns. And wherever possible, lean toward person-to-person communication. You’re changing hearts and minds, not reviewing governance policy.

    Third: Unite Your Team by Confronting Obstacles Together

    Don’t perform a shotgun wedding between ops and dev. Administrators and developers are drawn to their technology foci for personal reasons and interests. One of the most cited reasons for unsuccessful DevOps plans is a directive to homogenize the team, followed by shock this didn’t work. Developers are attracted to and rewarded for innovation and building new things, while admins take pride in finding ways to migrate the mission-critical apps everyone forgets about onto new hosting platforms. They’re complementary, integrable engineers, but they’re not interchangeable cogs.

    Contrary to popular opinion, telling developers they’re going to carry a pager for escalation doesn’t magically improve code quality and can slow innovation. They may even quit, even in this chaotic economy. And telling ops they need to learn code patterns, git merge and dev toolchains will be an unwelcome distraction not related to keeping their business running or meeting their personal review goals. They also may quit.

    It might be helpful to share with your team the simple idea you embrace: Unfounded stories of friction between dev and ops aren’t about the teams. They’re usually someone else’s war story about the organization, technology or process obstacles dividing them. At its heart, DevOps is about clearing these obstacles, not replacing them with new ones.

    By 2020, you’ve likely met someone from a team who had an opportunity to work based on tenets of DevOps, and they likely have no intention of going back. They’ll tell you they built or modified interaction models and evolved their deployment methodology organically. They streamlined the way code is built, improved testing at the bench level, staging and production. Even some companies needing to maintain specific waterfall processes for business or regulatory reasons may benefit if the transition is organic and incorporates their specific needs.

    Fourth: Set Your Plan in Action

    The last step—and this time, hopefully, the easiest—is to set your plan in motion, especially if you’re doing so at a smaller scale with better instrumentation than before. Ops teams and developers are people of action. We like to research, test and discover, but we’re happiest when our work sees production and we have a positive impact on the business. And often with initial forays into DevOps, stymied progress is what makes teams give up. By tracking ops and business metrics together, all the stakeholders will be able to share a sense of progress toward the overall goals, even when you break a few things along the way. Think of it as fail fast insurance.

    For example, consider adding development metrics and business metrics alongside your traditional infrastructure data. Track rate of change, error rate over time, deployment events and PR requests to get a measure of release times. Share your application performance management (APM) metrics with the whole team—real-time user-to-app-code-to-infrastructure-to-resources data sparks insightful cross-team conversation. But you should also track metrics affected by ops quality, including visitor time on site, page abandons, service desk ticket rate and CSAT. You likely already have the data you need, but perhaps it’s siloed in different systems. If your dashboards combine data mirroring the mix of teams and infrastructure flying in formation, it’ll be much easier to identify issues, evolve your initial plan to suit your unique environment and improve overall DevOps collaboration.

    And this time, you’ll be able to see code check-ins, unit tests, builds, pipeline progress, deployment to staging and prod, and follow-through on performance, issues and anomalies. If the team gets nervous trying something new, you’ll have actionable data, the production feedback loops developers crave and common charts everyone agrees on. It’ll be easier to know your DevOps effort is working and to keep pushing forward as you address unexpected challenges.

    You’ll Know It When You See It

    I made a joke in a talk once that DevOps is a lot like being in a relationship: You kind of fall into it. You learn about it, adopt some of its practices and enjoy the experience when it works. You don’t set out to “get in love” and likewise don’t set out to “get in DevOps.” Rather, you find yourself in a situation developed with experience over time, and you wake up one day and realize, “Hey, we’re a DevOps shop!” That’s why you can’t buy it in a box, and nobody else’s will be like yours. It’s not about the technology, it’s about the humans working together to manage it.

    Maybe this is the best reason to give DevOps another shot. It didn’t spring from a consulting team as a framework to drive engagement or from a vendor with an appliance requiring admins to bend to arcane workflows. It came from people; from developers and administrators who fought the same operations challenges we all have and who wanted to share what they learned. And this is why, outside of a bit of time investment from your team, DevOps is free. It’s made to help, not to sell you anything.

  • Is Your Infrastructure Ready for Edge Computing?

    Is Your Infrastructure Ready for Edge Computing?

    Like many IT innovations, edge computing began with engineers as a natural extension of technology to address a growing need. The concept isn’t new; distributed computing has been around for decades. But, at the same time standards began to converge and edge hardware started making the rounds at trade shows, the hype machine saw an opportunity. It amplified edge’s considerable promise in reducing latency, offering software-defined deployment, decreasing cloud networking costs and more. But as is too often the case, the bold feature bullets ignored the production concerns businesses must address, including edge computing’s rough spots and the additional operations complexity it adds.

    Of course, edge computing will survive a little overexcited promotion, just like many of the once improbable technologies before it. People used to say, “What? Abstract all my data center applications away from the hardware as virtual servers? Impossible!” A decade later, we can’t imagine how we’d deliver traditional enterprise services, cloud computing, online retail, media streaming and everything else in between without exactly this. Virtualization survived its awkward hype adolescence, and edge computing will, too. The needs edge computing addresses are only growing.

    Thanks to engineers and operations teams, the edge distributed model is moving toward practical use. It’s proving itself capable of meeting requirements for new levels of network performance through reduced latency, scalability and, more importantly, manageability. For some businesses, it’s even reducing costs over the long haul.

    Not All Data Is Created Equal

    With the proliferation of connected devices and a growing focus on 5G-enabled technology, tech pros should set aside their natural reluctance to wade through the edge hype and consider it a genuine possibility. Its adoption is following the rise of emerging technologies and the applications taking best advantage of it: 5G, augmented reality, autonomous vehicles, IoT and smart manufacturing. These environments require not only low upstream latency, but high-performance compute and timely result data. Light only travels so fast, pushing infrastructure closer and closer to consumers for faster, more seamless processing in the form of brand-differentiating user experiences.

  • DevOps for the Rest of Ops

    DevOps for the Rest of Ops

    The IT pro gig usually includes a basic headwind. They’re part of a cost center, and budget for resources and staff are limited by design unless they’re managing a strategic initiative, which—at least for a while—is more generously funded.

    Early adopters of DevOps, Agile or site reliability engineering (SRE) teams have been budget beneficiaries when tied to transformation projects—as has always been the case for high-visibility IT initiatives. Fortunately, DevOps has proved more helpful and stickier than some other “next big things,” and for some organizations it’s the engine behind growth and even actual digital transformation success.

    IT pros—pragmatic as always—have taken note of DevOps potential, even while keeping lights on, systems up and reliably delivering the critical services businesses have depended on for decades. The question for IT leaders is, how can IT teams realize some of the benefits of DevOps, while operating inside traditional IT organizations?

    As DevOps has finally slid off the hype peak and into the shed of useful tools, experience has distilled out a couple of conclusions. First, the tenets of DevOps—breaking down walls, shared feedback loops and learning to love automation—don’t only apply to new or cool apps. They’re even more helpful in traditional IT. For example, the rise of automation isn’t driven by a conspiracy to replace admins with robots, it’s a necessity to shrink the toil of manually wrangling all the new complexity of cloud and cloud-native tech on-premises.

    Second, many of the ideas behind DevOps and resulting practices are a direct attempt to fix issues in IT that have frustrated IT teams for years. Ask any IT pro if they’d like to move all deployment to Monday through Thursday, 9 a.m. to 4 p.m., instead of midnight Saturday nights. That team will go nuts for CI/CD. Better, it turns out you don’t have to do all the DevOps things to get many of its advantages.

    But even if your business still operates from the C-suite down, entraining quarterly budget spending, waterfall models and lots of walled silos, there’s an opportunity to adapt DevOps processes and culture to deliver progress on truly transformative initiatives to help grow the business. The trick is figuring out which DevOps practices are valuable and which ones are more useful to developers.

    Luckily, focusing on just four can get your team most of the benefits without (much) disruption, and development teams are eager to share its growing body of new techniques and successes. We’re working in an interesting epoch of IT history, a potential democratization of DevOps for everyone, DevOps for the rest of ops.

    Continuous Delivery

    Until recently, some IT pros were skeptical, even suspicious that the development teams’ goal has been to automate them out of a career. Others felt developers were demanding everyone to learn how to program full-stack and invent a way to move fast and break things just as they can (more safely) do. A few admins have even had unpleasant encounters with some DevOps proponents when they showed interest. Nobody likes reflexive replies, such as “You use Oracle and DB2? Switch to MongoDB and come back later” to eager 101 questions, even if they’re just from a handful of community members. It’s especially unpleasant for developers and IT pros alike because that’s not how operations works. We share, we’re open, we’re helpful. We came to tech to help end users get more out of technology.

    Instead, successful teams often start by identifying tools supporting fundamental operations goals, without the distraction of any particular technology or vendor, and seek buy-in from both developers and ops. Because if you’re in operations, you’re the team that must:

    1. Find a way to get software into reliable production.
    2. Maintain functional, useful governance.
    3. Maintain apps for years more than anyone thinks they’ll possibly be online.

    Operations knows this because it has always been the last mile of delivery and deployment, it cares about availability and SLAs, and defines process for breakfast.

    One of the first tools associated with DevOps is CI/CD pipelines, which are literally implementations of automated policy about how code goes from origination to production. But, overlooked in many DevOps blogs is the last step of CD because it’s not for the faint of heart. That’s unfortunate—operations is good at figuring out that last mile.

    When operations teams embrace CI/CD/Continuous Deployment, they’re rewarded for investment in automation in the best possible way: less night and weekend work. As a result, the pipelines they build aren’t ephemeral experiments but production-grade CI/CD because it’s part of production itself. And as change velocity increases and error rates decrease, many teams refuse to go back to manual deployment unless they’ve exhausted all other options. Pipelines don’t replace IT pros, they improve work-life balance.

    When the operations teams present developers with standardized, battle-tested nozzles through which changes get more quickly deployed with fewer errors, the reaction is usually excitement. It allows developers to mate their build, unit test, packaging and delivery head ends to the typically complex delivery/deployment operations pipeline. Best of all, ops teams tend to include monitoring, security and even cost governance into their processes, resulting in effective controls being automatically and reliably applied. Further, ops metrics create and enrich the feedback loops dev need to improve code over time based on real-world performance, not dev or test data. In environments like that, culture change is more organic based on shared goals, success and the flexibility to let each team do what they enjoy most.

    Start Breaking Stuff Fast

    Operations teams can have a little envy when it comes to breaking things. They’re technologists just like developers, and like most people attracted to technology, they like to learn by disassembling, recombining and otherwise hacking tech to learn the intricate details of how it works. But unlike a developer testing code, experimental configurations or other normal laptop environment one-offs, operations absolutely can’t experiment. IT is risk-averse by commandment, for good reasons.

    “Get a lab!” sounds like an obvious suggestion, and it is—but it’s a great example of easier said than done. Because IT is a cost center, labs are often seen as duplication and waste because their value is harder to justify in a budget presentation. But with a little unconventional use of the monitoring data you already have, that’s not necessarily true. Operations regularly correlates change events with performance histograms in the ongoing search for improved reliability. Consider adding before and after metrics to changes tested in a lab environment first. You’ll likely find you can prove the more realistic the pre-deployment testing, the lower the error rate and higher the performance of that system.

    But the mechanism behind improved quality isn’t the lab itself or data it produces. It’s the team’s experimentation—and some blood, sweat and tears shed—impossible in a production environment. IT also allows operations to invite developers to go on deployment and tuning ride-alongs. Developers can use whatever tools they prefer, make changes, probe and otherwise disassemble the lab stack as needed. And from a communications and culture perspective, this can be helpful.

    Operations lab environments also allow IT pros to separate hype from reality on their own. They can install the latest shiny new tools for evaluation, try new methodology or even test significant platform architecture redesigns. In short, they can break whatever they want without affecting production and provide both feedback to developers and a place for dev to experiment with changes. Here again operations can be the lead, with new avenues for communication, cooperation and culture change.

    Monitor Your Success

    Most developers aren’t data driven, they’re functionally driven. They love data-informing design and are good with tools that wrangle it into something useful, but at the end of the day their job is to build code that does things. IT pros have monitoring in the DNA because detailed performance data collected over time is how they initially validate hunches but also break down conclusion bias. But while everyone seems to understand the value of metrics and data, it’s not usually considered a tool for DevOps communication.

    As mentioned earlier, feedback loops are a fundamental principle of DevOps. Changes go into production, production generates data sent back to developers to guide future changes to further improve service quality. It’s a virtuous cycle, and both teams value it highly. The trick is in creating a language between dev and ops about what to collect, how it’s used and the stewardship of feedback over time. Step one is to sweep away preconceptions on both sides and start fresh.

    A common approach is to bring a small—one-pizza small—team together and start with a blank whiteboard. The goal is to matrix the inventory of what’s already available with what’s helpful, then identify gaps for both teams. Development teams are hungry for the incredible richness of coverage operations can offer, while operations is eager to mine new instrumentation the development team relies on for troubleshooting and internal application performance monitoring (APM).

    APM for hybrid applications is a great example of both teams learning how to create ensemble visibility into hybrid on-prem/cloud-native services. High performing operations teams ensure they add metrics dev teams need, and developers discover that the “impossible” information to debug pesky apps has been collected for years and is an API call away. Once again, a little conversation can lead to relief, and earned changes to hearts and minds.

    Reward Success With Big Dreams

    The final common experience of developers and IT pros both are never really done. Code can always be improved, and service delivery can always get better. But because we’re human, sometimes we get tired, stuck or bored, and it’s easy to take a break, especially after a big win. That’s natural. There are years of deferred work and technical debt lying around, and no time for perfectionism, especially in enterprise IT. This critical aspect of DevOps needs to remain a focus of any team adopting DevOps techniques—remembering DevOps is never done. It’s a culture of continuous, collaborative improvement and evolution. This shared understanding sustains the bonds between dev and ops.

    When teams innovate, when they continue forging closer relationships and when they share improvements as the team culture changes, they start asking bigger questions. We’re conditioned in IT to accept that certain things can’t be achieved because, reasons. As a result, sometimes suggestions and questions that could finally resolve the team’s most bedeviling challenges might not always be made or asked.

    Track your “impossible” ideas which drive progress, then promote successful ones while sharing details of why they worked. This in turn helps socialize the value of success resulting from openness and encourages great ideas that may have been sitting on the bench for a long time.

    Again, you’re investing in more metrics and signals, but not limited to performance data. When you dream big, especially as you untangle the enormous expectations of digital transformation into operations changes, you’ll start to rely on new metrics. For example, business metrics tied to corporate objectives, behavior signals accurately reflecting CSAT, change and error rate data helping teams decide when to add new versus when to fix and new ways to assure leadership operations have everything under control. When managers are confident IT pros can safely transform gear, services and staff without interrupting the business, the most sought-after DevOps goal can come into focus: A new freedom to innovate at speed.

    — Patrick Hubbard