devopsy.kinsta.cloud

Tag: engineering

  • The Curious Connection Between Cloud Repatriation and SRE Ops

    The Curious Connection Between Cloud Repatriation and SRE Ops

    I have a fondness for philosophy. I’m about three classes short of a degree, and every few years I tell myself one day I’ll finish it. Thus, I am very familiar with what is known in statistics—and logic—as a post hoc fallacy, from which we get the saying “correlation is not causation.” This is the logical error of assuming that if event Y followed event X, event Y must have been caused by event X. The most famous callout of this fallacy came from Bobby Henderson, who illustrated the absurdity of assuming causation from correlation with his chart demonstrating that global warming was caused by the declining number of pirates in the world. Yeah, that doesn’t make sense, but neither do a lot of charts from which people draw causality. Just because two data points are mapped against each other does not mean one caused the other. In many cases, it doesn’t even make logical sense to correlate the two. After all, pirates and global warming? No one actually takes that seriously. But it’s an important point to make as we dive into the question of the relationship between site reliability engineering (SRE) operations and cloud repatriation.

    To be clear, I’m not suggesting that adopting SRE practices causes cloud repatriation. But I am suggesting that there is a close and meaningful relationship between the two. The fact that Google—a cloud provider—created SRE as a practice is not a mistake. The model, mindset and skillsets associated with SRE are integral to successfully operating cloud infrastructure and services.

    Cloud Repatriation Is Real

    Public cloud repatriation itself is a somewhat taboo topic in certain circles. Consider the controversy raised by Andreessen Horowitz when it published “The Cost of Cloud, a Trillion Dollar Paradox” and suggested companies were repatriating from the cloud and realizing significant cost savings as a result. Some would have you believe it’s not happening, but there’s enough data and anecdotal evidence to indicate that yes, it is.

    F5’s 2021 state of application strategy report asked the market about public cloud repatriation. A mere 13% had repatriated apps and another 14% were planning to. One year later, that combined total rose 40 percentage points to 37% and 30%, respectively. This is not an anomaly, as there are multiple credible analyst firms reporting similar results. Interestingly, the rate of repatriation is not globally universal. APCJ and LATAM are both far less likely to repatriate than EMEA and NA.

    I maintain that companies are repatriating apps from the public cloud—and the question isn’t ‘Are they?’ but rather ‘How many workloads are they pulling out—and where are they going?’ That’s a question for the State of Application Strategy 2023 research.

    For now, let’s consider a possible enabler of repatriation—SRE operations. Because even if the increasing cost of cloud is a driver of the desire to repatriate if you don’t have the skills to operate as efficiently elsewhere—and thus benefit from lower cost—then why would you repatriate?

    And we posit that it is SRE operational practices and skills that enable companies to repatriate and maintain the efficiency and cost savings needed to justify the decision—whether they’re moving those workloads to another public cloud, on-premises or to the edge.

    Digging Into the Data

    On the surface, there is a strong correlation between the adoption and application of SRE practices with cloud repatriation that seems to indicate that organizations with the ability to operate in a cloud-like manner, i.e., they’ve adopted SRE practices, effectively pick up their toys (apps) and go home (on-premises or elsewhere) because they can.

    Cut another way, only 4% of organizations that have not adopted SRE practices have repatriated apps from the public cloud. A whopping 73% of those who have adopted SRE practices have also repatriated apps.

    Of course, adopting practices does not necessarily mean applying practices. So, we looked at how organizations are actually operating applications, systems, and infrastructure. Specifically, we looked at the percentage of their operations that use SRE practices. Perhaps unsurprisingly, that generated similar results.

    Of those who operate 0% of their apps, systems, and infrastructure using SRE practices, 81% are not repatriating. Conversely, of those who use SRE practices for 76%–99% of apps, systems and infrastructure operations, 54% have repatriated. The point at which repatriation appears to begin picking up steam is when organizations surpass using SRE practices to operate more than one-quarter (25%) of their apps, systems and infrastructure.

    Remember I noted that APCJ and LATAM were far less likely to repatriate? They’re also far less likely to be leveraging SRE practices to operate their apps, systems, and infrastructure. In fact, over one-quarter (26%) of respondents in LATAM and APCJ (29%) were operating ZERO percent of apps, systems and infrastructure using SRE practices. In EMEA? That’s only 5%. And in NA, even lower at 2%.

    Meaningful Relationship or Curious Coincidence?

    There appears to be an inarguable correlation between organizations embracing SRE as an operational practice and public cloud repatriation rates. But is it a meaningful relationship or merely a curious coincidence?

    I’m going to argue that it’s a meaningful relationship.

    The practices and skillsets associated with SRE are wholly suited to operating a cloudy environment—at scale. As I said before, it’s no mistake that it was Google who created SRE and has literally written the book on it. The value of cloud is in its operational model, which can dramatically lower the cost per transaction, whether measured by HTTP exchanges or customer sessions. That enables a cost-efficient scale of applications and digital services.

    The use of automation and practices that tend to focus on meaningful incidents rather than non-disruptive occurrences provides the cost-efficient scale of the people (and thus their expertise) who are tasked with maintaining a high level of availability and performance.

    The adoption and use of SRE practices enable organizations to efficiently scale operations whether in the public cloud or on-premises or at the edge. And what the data tells us is that organizations appear to be using the capability to do just that.

  • How to Manage and Engage Remote Engineering Teams

    How to Manage and Engage Remote Engineering Teams

    As engineering teams look to remain remote or hybrid, executives need to fundamentally change how these teams are managed. After tackling the tactical aspects of a distributed workforce—like home office equipment, a comfortable space where employees can get work done and reliable meeting, calendar and communication tools—more nuanced issues come into play.

    Before the COVID-19 pandemic, when a minority of engineering team members worked remotely, those team members knew that they needed to make a concerted effort to connect and to be part of the group. When the majority are remote, however, the expectations of offsite employees have shifted dramatically. For organizations that had built their business with an in-person team, this brings up entirely new questions of communication, culture, loyalty, problem-solving and retention to the forefront.

    Fundamentally Shifting Engineering Team Structure

    Inspiring change requires fundamental shifts in the ways engineering teams work. Our own engineering teams have experimented with some changes that have proven effective, including:

    • A globally revised team structure that embraces a culture of autonomy and breaks larger engineering teams into smaller groups. Smaller groups help engineers feel more engaged, valued and empowered; it keeps meetings short and focused and allows for more parallel streams of work.
    • Overhauling our design process to get a champion or ‘anchor’ from the dev team involved in key features early on. This anchor is responsible for translating product and design requests into technical specs and is the de facto technical lead of the feature. Having this role rotate between team members for each feature means each member gets a chance to grow their skills as a technical lead.
    • Assigning dedicated designers to smaller groups to empower them to make small adjustments mid-flight. This would allow standups to operate as mini working sessions to iron out technical details that would normally be resolved through ‘shoulder taps’ or in-person quick questions.
    • Placing a strong emphasis on sprint retros as a way to make real changes to the way work gets done. When used in conjunction with an initiative to empower devs, sprint retros became a powerful tool to get dev feedback on improving the process.
    • Keeping teams accountable by encouraging transparency and clear communication as a tradeoff for their autonomy. We encouraged note-taking and regular updates, as well as clear goal-setting from the teams at the beginning of each sprint. By holding teams accountable to the goals that they set, it became easy to see when things were and were not working.
    • Experimenting with a four-day workweek to see if we can drive both greater happiness and productivity. This was perhaps the boldest of our experiments; we demonstrated trust in our team and set clear expectations around outcomes rather than hours. This has motivated our team and raised their morale in crucial ways at a time when switching jobs was both tempting and easy.

    Hacks That Worked in Person

    What are some of the things that worked in the office? This list will be different for each organization; for us, it was group brainstorms, in-person one-on-ones, summer Fridays and ongoing vibe checks. Here are some of the ways we hacked these to make them work for remote teams:

    • Group brainstorms: Digital whiteboards like Miro or Mural have been great. Digital sticky notes, too. This has enabled distributed engineering teams to contribute either all at once or asynchronously.
    • Summer Fridays: Give people the afternoon off even if it isn’t technically “leaving the office early.” Or try having everyone shut off notifications on Friday (or Monday morning) so they can get some undisturbed work done before shutting off for the weekend. Monthly mental health days off for the whole company can be a great idea too.
    • In-person one-on-ones: With Zoom, something is definitely lacking. Try spending half the one-on-one meeting chatting casually about things other than work.
    • Vibe check: Open office hours dedicated to work culture; surveys from time to time to gather quick informal feedback or think about other ways to elicit ongoing feedback from every employee, including engineering teams.

    Generate Culture and Well-Being

    What keeps most employees at companies is social bonding, inspiring projects and the possibility for growth. A question a lot of executives at previously onsite organizations are asking themselves is, “How do you fulfill the need for social bonding when most employees, including your engineering team, are remote?” In other words, how do you bond socially and do so remotely? Two goals we have identified are getting team members talking outside of work while avoiding Zoom fatigue issues and ongoing opportunities to share personal successes with colleagues.

    Some of the ways we have built culture are:

    • Having (non-mandatory) Friday lunch and games like trivia, Jeopardy!, games or online games
    • Different ongoing events like photography challenges, health challenges and even an at-home Iron Chef-style cookoffs.
    • Having leaders share how they spent their weekends and time off. We believe in demonstrating a healthy work-life balance from the top down and encouraging our team to take the time they need to unplug and recharge.
    • Figuring out how (and how not) to surface urgent issues via Slack; working on a system to avoid pinging people after hours for non-urgent issues.

    Above All, Manage With Empathy

    These last two years have been exhausting for most people. Hands down, the best way to manage and engage the people you work with—no matter if they are in engineering or marketing—is to be empathetic to the difficulties that they may be going through outside of work. Executive teams can make all the tactical changes to their teams they want, but without managers that really care about their teams, you will inevitably fail to keep the best people around. Empathy-driven leaders lift their team members up and drive the kind of culture people want to be part of.

  • Report Finds Software Engineering Productivity Gains

    Report Finds Software Engineering Productivity Gains

    An analysis of the time 23,000 software engineers spent working in 2021 that was published this week by Jellyfish, a provider of an engineering management platform (EMP), finds cycle times on average were down to 12.6 days, representing a 12.3% decline over 2020. The report, however, also finds commits were down 4% year-over-year, while the number of issues resolved was up 5% year over year.

    Jellyfish CEO Andrew Lau said the data suggests that software engineers are opting to commit less code in favor of resolving issues faster. Overall, coding days per engineer were down 8% to less than 1.6 days per week in 2021, the report found.

    Engineering teams on average in 2021 allocated 35% of their time to growth and innovation, while 19% was allocated to infrastructure work, up nearly 6% from 2020. Time spent on support work (24%), meanwhile, was down nearly 5% year over year. On average, unplanned work accounted for 22% of the time software engineers spent working in 2021, a 3% increase over the prior year. In general, Lau said the report suggests that engineering teams need to factor more time into schedules for unplanned work as application environments become more complex.

    The report also finds engineering organizations using EMPs tend to allocate similar time to “keeping the lights on” and customer support activities as other organizations but spend 29% more time on innovation work and 48% less time on unplanned work. In addition, the report notes teams that devote 50% or more resources to growth and innovation work deliver software at least three times faster than those who commit 25% or less.

    Virtual collaboration in the last year has also increased more than 20% as teams continue to embrace remote and hybrid work models, according to the report. Overall, organizations that allow software engineers to work from home remain productive but it’s not quite clear if managers have completely adjusted to the realities of hybrid work, noted Lau. The report also noted that pull request reviews per engineer are up 30% on average to 1.7 reviews per week. Pull review comments per engineer were up 23% year over year.

    In addition to the report, Jellyfish conducted a survey of 128 engineering leaders that finds more than 79% expect to grow their teams in 2022, with 57% expecting more than 10% headcount growth. Improving operations, tooling and processes (36%) is their highest priority in 2022, followed by hiring and retaining talent (18%), improving infrastructure (17%) and managing technical debt (10%). The biggest overall challenge is making sure everyone is focused on the highest priority work (45%), the survey finds.

    However, only 27% of the leaders currently track productivity metrics, despite more than two-thirds (67%) describing them as being very important or critical to track. Among teams that do track metrics, more than half (53%) do so manually, the survey finds.

    While the productivity of software engineers will naturally vary by organization, it’s apparent the weight of application environments is starting to impact how much time can be devoted to building and deploying new applications. The challenge, of course, is demand for new applications in the age of digital transformation has never been higher, which in part explains why so many organizations are struggling to find and retain software engineers.

  • Prioritizing Scalability, Reliability and Security in Engineering

    Prioritizing Scalability, Reliability and Security in Engineering

    As digital products and services are more deeply embedded in critical industries and infrastructure and the implications of problems grow in scope, engineering organizations are renewing their focus on building platforms that are scalable, reliable and secure. More importantly, they are recognizing that this isn’t the responsibility of an SRE or CSO, it’s an ethos that must be carried each day by every engineer on the team. This is an important shift—from teams and cultures that valued speed of delivery above all else to teams and cultures that make high-quality systems the foundation of everything they do.

    This isn’t easy, and big cultural changes don’t happen overnight. There are many things that can go wrong when the team is operating in the legacy mode of prioritizing quick time-to-market but there are also some strategies and best practices for beginning the process of cultural change across your engineering organization.

    The Old Way of Doing Things

    You’re the engineer on call, and you get woken up at 3:00 a.m. by an alert that tells you an app is down. Groggy and annoyed, you look at the message and have no idea what service it’s referring to or where to begin.

    After nearly an hour, you can’t figure out who owns the service or how to get it up and running. So, you call that senior engineer on your team—you know the one—the person who gets called every time there’s a tricky question or problem. Thankfully, they answer your call in the middle of the night and help you bring the app back up. But in a month, the same scenario happens again; only this time, when you go to call that senior engineer, they’re no longer there. They quit to go work at a company that cares about reliability.

    Best Practices for Today

    How can you avoid this vicious cycle? Here are some best practices you can implement to begin shifting focus away from speed and toward scalability, reliability and security.

    1. Evangelize the Narrative That the Systems you Work on Matter

    At the end of the day, your systems are there to serve your customers and users. Leadership and teammates who embody these values are incredibly important. People should worry about letting their customers down, not about being yelled at by their manager or receiving a bad performance review.

    2. Set Explicit Expectations about Uptime

    When you say that your app needs to have 99.99% uptime, it doesn’t have the same impact as saying 52.6 minutes of downtime a year. Translate uptime numbers so they’re tangible—and on people’s minds every time there’s an outage.

    3. Don’t Sweep Problems Under the Rug

    After the dust settles from an incident, it’s important to get together and talk about it. Be transparent about what happened, which person or team made the error and what the implications to users were. The idea isn’t to point fingers or assign blame but to bring the team together so that you can learn, evolve and figure out how to prevent this from happening in the future.

    Here are some questions you can use:

    1. Who wrote the code?
    2. How was it tested?
    3. Should our automated testing have caught this?
    4. Who reviewed this code?
    5. Were they the right people to review the code?
    6. Why wasn’t it caught in the staging environment?
    7. Who released it and who went through the release validation checklist?

    Formalizing this process lets everyone on the team know that mistakes are part of the job and that there’s a standard way that problems are handled. This is a great way to help engineers feel free to do their best work without fear or avoidance.

    4. Define Individual Owners for Every Piece of the System

    If there’s any confusion about who owns what within your engineering org, you need to address that right away; it’s a massive problem. Not only do you need an owner for every part of the system, but they must also be empowered to fix issues and know that they’re the ones on the hook.

    5. Invest in Documentation, Runbooks and Logs

    The owner of each piece of the application is also responsible for making sure that documentation is up to date, runbooks are clear and easy to follow and logs are readily accessible. When you’re getting paged at 3:00 a.m. about a service you’ve never heard of before, these resources are key.

    6. Prioritize the Delivery Pipeline Like it’s a Key Feature

    You should always be thinking about how to build safeguards around your workflows, like automated testing or new tools. This may mean you need to go toe-to-toe with product managers to make sure the work required to build a robust delivery system isn’t getting kicked down to the backlog.

    7. Get Business Buy-In by Using Terms They Understand

    Unreliable code is technical debt. And debt is something all leaders understand, regardless of their technical abilities. Stripe published a survey in 2018 that found the average developer spends more than 17 hours a week dealing with maintenance issues, costing $300 billion per year globally. If you’re pushing to invest in SRE practices, those numbers might help you make your case.

    Culture comes first, but the right tools can reinforce the culture and drive accountability over time. Tools that help teams keep track of critical information such as service owners, documentation and runbooks can help tremendously. Additionally, having easily available dependency maps of the architecture are key to quickly understanding complex systems. Lastly, setting organization-wide standards for best practices and providing templates for spinning up new services that follow these can help ensure that these things are not forgotten.

    This article was based on a talk given by the author at Conf42: SRE 2021. 

  • Why Boring Tech is Best to Avoid a Microservices Mess

    Why Boring Tech is Best to Avoid a Microservices Mess

    Microservices are all the rage. A Forrester study found that 76% of enterprises were rearchitecting applications for microservices.

    At the same time, microservices are definitely not a silver bullet. Among those already using microservices in production, one study found that 59% said each microservice added increased operational challenges such as data management. Even more concerning, the same report revealed that when asked to compare the difficulty of troubleshooting different environments, 73% said microservices were more difficult, and only 21% said they were easier than monoliths.

    But how did we end up like this? According to Matthew Sinclair, VP of engineering at BCG Digital Ventures, it is because that, as technologists, “we want to use the most modern tech, and sometimes we even want to do it with tech we’ve got no experience in. We see this with engineers, not because it’s the right thing to do but because they want to learn something new.”

    Sinclair, who was speaking as a part of an Indorse Engineering Leaders Online Panel, went on to point out how the problem becomes self-perpetuating: Customers hear about new tech—in this instance microservices—that will fix the problems they have with monoliths. They rave about microservices to software engineers, who also get excited about it and then go off and start adding microservices in software engineering projects.

    The problem is, there is just as much chance of getting into a tangle with microservices as there is with monoliths. That’s because people often start breaking things up into microservices too early—they’ve heard it will be a good idea and they’ve seen some of the benefits, so they try to apply it to everything. In doing so, they create what some might call a “micromess.”

    Avoiding the Micromess

    So, how do you avoid ending up in a micromess?

    You need to go back to the fundamentals. One way to look at it is understanding that microservices are distributed systems, something many people will have experience with. They should also be familiar with what another panelist, Oswaldo Hernandez Perez, engineering manager at Improbable, called the first law of distribution: “If you don’t need to do it, don’t do it.”

    So that means focusing on why you are building what you are building. What are you trying to achieve? This is a fundamental question that’s applicable to businesses of all sizes. What problem are you trying to solve, and how will your solution remove friction from its users’ lives?

    That’s what people care about. Even if you’re developing a niche app for a highly technical audience, they are unlikely to care too much about how it got to them, only that it did and it is fixing a problem for them.

    If the only way to achieve that is with microservices, then yes, you should definitely use them. If there is an alternative, then consider using that. Do not simply start breaking everything up into microservices just because that is what everyone is currently talking about.

    Don’t Stick Your Head in the Micromess

    Ultimately, microservices are an architectural pattern to reduce complexity. It does this, but it also adds complexity elsewhere. If used in isolation, then you’ll fix your complexity in one dimension and have it proliferate elsewhere. So, it is vital that you use the many other tools out there to minimize complexity across the entire organization.

    It goes back to the question: What problem are you trying to fix? The majority of businesses can do that with boring tech. Granted, the likes of Amazon and Google, which are cloud-native to their core, can use microservices, but even large enterprises should be asking themselves if it’s really the right choice for them.

    Established businesses are likely to have a significant legacy codebase, which needs integrating with. Again, microservices might not be appropriate in this situation—it might be worth saving them for cloud-native projects, and use boring tech to resolve the challenges within legacy IT.

    Do You Really Need Microservices?

    Time and time again, we come back to the fundamental question: What are you trying to fix with this product and/or technology? Not how, but what? What friction will you banish from the user’s life if they engage with your product? Only once you answer that question should you consider whether microservices is for you. Chances are, you may find you don’t need them by that point.

  • Best of 2019: The Time Has Come for an Engineering Approach to DevOps

    Best of 2019: The Time Has Come for an Engineering Approach to DevOps

    As we close out 2019, we at devopsy.kinsta.cloud wanted to highlight the five most popular articles of the year. Following is the third in our weeklong series of the Best of 2019.

    Do you want to know how to do DevOps? Are you doing DevOps but not satisfied with the results you are getting? DevOps is complex. It does not come with a prescription, installation guide, user’s manual or maintenance manual. It doesn’t even have a standard definition. How can anyone expect to get good results with DevOps without clear and definitive guidance? There are plenty of books that describe different aspects of DevOps and customer user stories, but up until now there has not been a book that frames DevOps as an engineering problem with a step-by-step engineering solution and a clear list of recommended engineering practices to guide implementers.

    My book, Engineering DevOps: From Chaos to Continuous Improvement … and Beyond, provides a step-by-step engineering prescription that can be followed by leaders and practitioners to understand, assess, define, implement, operationalize and evolve DevOps for their organization. It provides a unique collection of engineering practices and solutions for DevOps. By confining the scope of the content of the book to the level of engineering practices, the content is applicable to the widest possible range of implementations.

    This book was born out of my desire to help others to do DevOps, combined with a burning personal frustration. The frustration comes from hearing leaders and practitioners say, “We think we are doing DevOps, but we are not getting the business results we had expected.”

    There are many documented references, courses and other sources of information that explain a multitude of different aspects of DevOps, but there is not a comprehensive step-by-step prescriptive guide that details how to engineer a DevOps solution for any organization given any starting point.

    Engineering DevOps takes a strategic approach, applies engineering implementation discipline and focuses operational expertise to define and accomplish specific goals for each leg of your DevOps journey. This book guides the reader through a journey from defining an engineering strategy for DevOps to implementing The Three Ways of DevOps maturity using engineering practices: The First Way (called “Continuous Flow”) to The Second Way (called “Continuous Feedback”) and finally The Third Way (called “Continuous Improvement”). This book is deliberately intended to be a guide that will continue to be relevant over time as your specific DevOps, and DevOps more generally, evolves.

    Structure of this book

    This book is organized as an engineering reference guide presented in five parts as follows:

    Part I: What Is Engineering DevOps and Why Is It Important? is organized into three chapters. Chapter one, “What is Engineering DevOps?,” explains engineering concepts and terms used in this book that have proven to work with many clients I have encountered during my consulting experiences. My depiction of a DevOps Engineering Blueprint is presented for as a useful, practical “big-picture” reference for discussing engineering the major parts of DevOps. Chapter two, “Nine Pillars of DevOps,” describes a classification of DevOps practices that I have found to be useful characterization of DevOps engineering practices readily applied when engineering DevOps implementations. Chapter three, “Why Is Engineering DevOps Important?,” explains the benefits of taking an engineering approach to engineering DevOps.

    Part II: Engineering People, Processes and Technologies for DevOps provides a comprehensive explanation of recommended engineering practices for the higher levels of the DevOps Engineering Blueprint. DevOps transformations are not normally understood or applied using strict engineering principles. Engineering practices are presented in nine chapters. “How Should DevOps Be Engineered?” includes a discussion of DevOps Engineering Maturity Levels for each of the Three Dimensions (People, Process, and Technology), Twenty-Seven DevOps Engineering Critical Success Factors and Lean DevOps Value-Stream Pipeline Engineering.

    I describe recommended engineering practices for the top layers of the DevOps Engineering Blueprint in the next chapters, which include “Value-Stream Management (VSM),” “Application Release Automation (ARA),” “Version Management,” “Continuous Security (a.k.a. DevSecOps),” “Service Catalog,” “Governance,” “Site Reliability Engineering (SRE),” “Disaster Mitigation” and “Recovery.”

    Part III: Engineering Applications, Pipelines, and Infrastructures Engineered for DevOps provides a comprehensive explanation of recommended engineering practices for the lower levels of the DevOps Engineering Blueprint, presented in six chapters as follows: “DevOps Applications Engineering,” “CI/CD Pipelines Engineering,” “Elastic Infrastructure Engineering,” “Continuous Test Engineering,” “Continuous Monitoring Engineering” and “Continuous Delivery and Deployment Engineering.”

    Part IV: DevOps Seven-Step Transformation Engineering Blueprint provides a description and tools for my approach to realize and evolve DevOps. The seven steps are Visioning, Alignment, Assessment, Solution Engineering, Realization, Operationalize and Expansion. By using these tools, DevOps leaders and practitioners can create, implement, operate and expand their DevOps across the organization. The chapter goes further to explain how to evolve DevOps from a successful First Way DevOps (Continuous Flow) toward realizing more advanced Second Way (Continuous Feedback) and Third Way (Continuous Improvement) DevOps implementations. This part includes a discussion of “Beyond Continuous Improvement”—a look at emerging technologies shaping DevOps in the future, and how you can prepare your DevOps and yourself for the future. This part includes a discussion of how to set up an effective DevOps engineering training program that supports continuous learning of DevOps engineering skills needed to maintain and enhance DevOps.

    Part V: Appendices, Continuous Learning, and References includes materials and sources I have found most useful for engineering DevOps.

    To get a copy of my book “Engineering DevOps: From Chaos to Continuous Improvement … and Beyond,” or start reading for free, click here.

    — Marc Hornbeek

  • How to Become a Great Engineering Manager

    How to Become a Great Engineering Manager

    I wrote recently about the attributes of a 10x software engineer and a 20x one. Consequently, I received a few comments suggesting I create a similar list for engineering managers. I checked online and found a few articles, but none conveyed a full list in my mind. If you consider that software development efficiency has become the No. 1 priority for executives, ranking higher than access to engineering talent or capital, then I guess that’s a topic worth discussing! Engineering managers just might have the biggest impact on the productivity of their engineering teams.

    I’ll first go over the attributes for a good engineering manager, and then list what makes a great one.

    The 10 Attributes of a Good Engineering Manager

    I tried to prioritize this list, from essential to bonus. Feel free to give me your opinion on this in the comments.

    1. Be trusting

    If you refer to what destroys developer productivity most, trust is at the heart of it. Managers need to trust their team and give them some space. Micromanaging means undermining the team’s skills and ability to get things done, and therefore their motivation and self-confidence (which, by the way, they need to address difficult problems). Trust between the managers and their teams should be the foundation of their relationship. And it’s the manager’s role and job to create it, not the other way around. So first, trust your team.

    1. Emotional intelligence and ability to establish trust

    This is linked to the first point. You need mutual trust within a team for it to work well. And again, it’s the manager’s job to ensure that their team trusts them. How does the manager do this? By showing emotional intelligence.

    Engineering managers must be sure to make themselves available for questions, even if they’d rather not be interrupted. This can run counter to high efficiency, but it’s important that managers spend the time to form personal connections, so honest and open conversations happen naturally within the team and with the manager.

    Another important point is to listen to anyone and everyone. When the team gets together for a meeting to sort out a project they plan to do, everyone in the room needs to be heard and considered. The manager needs to respect any experience or insight their team can bring to the table. Any member of the team may be the most knowledgeable on a particular subject.

    Now, this is not Disneyland for sure: Managers might need to be tough and unwavering sometimes. But when that’s the case, it’s important to be transparent about the reasons to keep the hard-earned trust intact.

    1. Ability to shield the team from interruptions

    Interruptions and meetings (consider them planned interruptions) are greatly loathed by developers, and for good reason. They literally destroy their productivity. Developers love projects that move fast; they’re invigorating! Managers would do them a great service by removing distractions, ensuring that they have large chunks of uninterrupted “maker time.”

    A classic example is a really long email exchange on some technical topic with no resolution. The manager could simply put everyone on the thread in one room together for 10 minutes to get the issue resolved promptly.

    1. Ability to manage the team’s motivation

    The team’s motivation is directly linked to their output and the success of their releases. There are many ways to keep the motivation high. Here are just a few:

    • Help the team see their work through the lens of their impact on the company.
    • Don’t worry too much about failure. Things will always fail: builds, deploys, schedules missed, etc. If things fail, take responsibility and focus on the learning. The point is not to have to fix the same issue twice. This will give your team a mindset toward action and growth.
    • Celebrate every achievement, whatever its size. And credit the participants profusely.
    • Be transparent. Default to overcommunicating. While some leaders believe they need to shield their teams from things going on at the executive level to prevent distraction, there’s a difference between productive shielding and unintentionally denying engineers important company context. Managers want their teams to feel included and invested in the direction of the company.
    1. Strong work ethic and cohesion with company culture

    Work ethic means being exemplary in all the ways you expect your team to behave. Lead by example. Managers must embrace the company culture, or it will create confusion sending team members mixed signals of whether or not they should embrace the company’s culture.

    This also means shielding your team from people who don’t act accordingly. For instance, hiring someone with a huge list of qualifications and experience can still be a disaster if they have a different set of values than those held by the company. Or, unfortunately, having to let people go who are not team players or aligned with the company and team culture, whatever their skill level.

    Good managers keep in mind that the team’s productivity as a group is important, not individual performance.

    1. Advocate for change to seek positive evolution in a constant manner

    Good managers install a mindset of positive evolution within the team.  They adopt a state of mind that continuously seeks to improve a certain situation. When a process is slow or cumbersome, they try to turn that around and make it better. One way todo this is using OODA loops: Observe, Orient, Decide, Act.

    1. Ability to work with departments outside of engineering

    The language used by non-technical stakeholders might be very different than that of the development team. However, good managers don’t want to be the Single point of contact, as they would introduce a potential liability into the project—a strong dependency on them. They tend to include their team in relevant discussions, but make sure they protect the team members from continuous interruptions. Good managers try to be the First point of contact, which is a big difference from being the single point of contact.

    1. Mentoring abilities

    Effective managers are measured by the effectiveness of their software teams. Anything that they can do to make each person on their team better makes their overall teams exponentially better. And most of the mentoring is actually about matchmaking team members with high-impact projects that align with their interests, the value they want to create and how they want to grow. That’s when motivation becomes organic.

    There are already so many articles on one-on-one meetings, I won’t spend much time on them. But mentoring also happens every day at every occasion. For instance, if a manager happens to review code or explain certain conventions, they make sure to clearly explain the reasoning why the team is doing something in a particular manner. Good managers are great teachers.

    1. Technical literacy and problem-solving skills

    On the one hand, technically competent managers will have a superior handle on the work that they are managing, and will therefore be better able to make recommendations and evaluate necessary trade-offs. On the other hand, technically talented managers are often better able to garner respect from their teams. Non-technical managers may struggle to manage high-performing individual contributors, who tend to have a tough time taking orders from managers who don’t fully understand their craft.

    But, if you don’t have any engineering background, it doesn’t mean you cannot be a good—or even a great—manager. I would advise you to learn about the technical dimension of the project (with humility) through every interaction you have with your team, and possibly in your off hours. Having enough technical experience to understand the projects and the decisions, mixed with the ability to trust others with technical depth, works really well.

    1. Ability to make hard decisions and manage conflicts

    Good engineering managers will make the courageous decision, even if it isn’t popular. They don’t make the wrong decision just because it is socially right and their team suggests it. Good engineering managers focus on the road and the values they believe in, and not only on the immediate issues they are facing. They don’t just choose best industry practices because they will attract the “right” talent. For instance, managers will avoid  automatically using the latest technologies: they will weigh the pros and cons and make the best choices based on the resources and requirements of their team and company.

    How Can You Become a Great Engineering Manager?

    1. Embody the vision

    Great managers have a natural ability to visualize the future of the company and motivate others to work toward those lofty yet realistic goals. They communicate the company vision at every opportunity by illustrating it contextually in discussions. They challenge the decisions made with, “What if we had 20x more users? Would that be sustainable?” or, “Will this technology decision still be valid in 24 or 36 months?”

    1. Support your team in the short and long term

    At the end of the day, as a technical manager, your job is mostly about humans. Great managers think about supporting their team in two ways: day to day and year to year.

    Great managers help their people devise a long-term plan to be happy and productive at their company. Sometimes, engineers don’t know how to articulate how they want to evolve. This is not an issue, as great managers have built strong relationships and trust with their team members to help them uncover this part and build a plan for:

    • the skills to improve.
    • the technical/non-technical experiences they want to have.
    • how they want to increase the scope of their impact at the company.

    And, naturally, the great managers will help their people to achieve their ambitions (within or maybe even outside the company).

    1. Challenge the team to grow faster

    Great managers try to super-charge learning within their team. They know to push their people out of their comfort zone and challenge them to accomplish things they didn’t even know they could do. But in doing so, they set up strong guardrails to help their team course-correct along the way.

    For instance, one way for engineers to increase their impact is to architect and own increasingly complex technical problems requiring novel solutions. Great managers will throw their people  into the deep end of the pool, with the right mentorship and guardrails, and have them start solving increasingly complex problems immediately. They will also try to maintain a balance between more experienced and less experienced engineers, having them work together and regularly share knowledge. They want to build in opportunities to make sure these conversations are truly bi-directional.

    Another instance is estimates. Do you know this law?

    Hofstadter’s Law: It always takes longer than you expect, even when you take into account Hofstadter’s Law.

    Estimating is hard. When you do it more often, you will get better at it … but you will still get it wrong from time to time. Challenging your team to make estimates is a way to get them used to getting out of their comfort zone.

    1. Embrace the differences

    Diversity is invaluable. All people are different and live different lives. This is incredibly valuable, because your users will also be different. Surround yourself with passionate people of different backgrounds. By background, I don’t only mean gender or race, but also education and past experiences. The more people in the team think differently, the more approaches your team will consider when addressing a problem, which results in better decisions. A great manager understands this and will strive to get input on the same problems from a variety of people with different backgrounds.

    1. Level up the product and engineering partnership

    Great engineering managers know to extract the essence from the user story; they focus first on the core capabilities, providing valuable input and challenging the product as early as possible. They never blame the product on complex features, or make the excuse that some things were not explained in the first place or changed 10 times. They never complain about the lack of visibility on the ideation process. They build such visibility themselves. They challenge the product team with several alternatives for the same idea. They know to explain pros and cons without deep diving into tech level and buzzwords. They help the product team make better decisions and educate them about the technical layers of features.

    1. Know themselves and educate others about themselves

    Great engineering managers have made the effort to understand how they themselves work,  behave,  think and make decisions. Being transparent with their team about these things, builds more trustful and productive relationships. They work on visibility and commitment for their own goals and tasks in front of their team members.

    1. Push their team for ownership as much as possible

    Great engineering managers tend to give more ownership of user stories to their developers. Transferring ownership to developers gives them more motivation to do a great job. Great engineering managers don’t try to own all of the responsibilities or give them all to a few select  individuals (tech leads).

    Great managers know when to give input, when to make decisions, and when to step back and allow the team to take more ownership. They promote a culture of accountability, where they trust each team member to do his or her job and never  insist on full visibility into what their team is doing.

    1. Understand the system’s bigger limitations

    Great managers understand the inefficiencies within the processes and organizations, as well as their own limitations. They share their thoughts to upper management to help the organization grow (and not only their team).

    This concludes my list of the attributes of good to great engineering managers. If you think of any I’ve overlooked or have any ideas or suggestions to add, I would love to hear your thoughts! Please leave your comments and opinions below!

    — John Lafleur

  • Cuba Embraces DevOps, Despite Infrastructure Challenges

    Cuba Embraces DevOps, Despite Infrastructure Challenges

    A few weeks ago I had the pleasure of speaking at the second DevOpsDays event in Cuba alongside some great technologists from across the United States, Europe and Latin America. The conference was held in a beautiful hotel in Varadero overlooking a marina right by the beach. Varadero is approximately two hours by car from Havana.

    The first thing I noticed is that Cuba is full of anachronisms. This became evident as soon as I walked out of Havana Airport and realized that the whole “they have some old cars” thing was real. It was fascinating to see that every single car was pre-1950s, some perfectly maintained with great paint jobs and others in worse condition yet still capable of doing their core job as a mode of transport.

    Talking technology in Cuba was eye-opening, to say the least. Do not expect to have 3G or 4G data on your cellphone. Do not expect to have WiFi in most hotels (you can purchase prepaid internet credit, but the speed will be slower than what you are used to) and definitely do not expect to have WiFi in a private home. These are luxuries.

    But here’s the surprising part: There are companies in Cuba with fully automated CI/CD pipelines achieving 10 to 12 deployments per day and solving real problems, such as the team working under Anesto del Toro at Datys. Cuba even has its very own Docker Captain, who gave a superb demonstration on the value of Docker in front of the DevOpsDays audience.

    Now let me remind you, these guys are not working with stable internet connections and they manage to get the job done on a daily basis.

    Why am I Rambling on About Cuba?

    Engineering is about solving problems. No matter how big or small, it’s about looking at a problem and finding the solution. Now, this is part of Cuban life. If something is broken, they don’t throw it in the trash and buy a new one; they fix it. Their 80-year-old car breaks down two hours outside of town? They get out of the car to fix it and—take my word for this—their friends, family and neighbors will come outside to fix the problem, too. This creates a mindset of solving the problem at hand without fail. This is easily demonstrated when you talk to Cubans about software development: They won’t say they are a Python engineer, a Java engineer or any other choice of language—they are software engineers who understand how development works as a whole. They will use the language required to get the job done to the highest standard.

    Like everywhere else in the world, there were huge differences in levels of DevOps adoption between the various organizations in the country. In contrast to the companies mentioned earlier, there also are organizations that have only just heard of these methodologies or are now considering setting up their first CI pipelines. Like many of us, they are still figuring out how to solve their technical debt and modernize their IT Infrastructure, but the key point is that the stakeholders, C-level executives, junior engineers and everyone else in between want to learn. They are open-minded to the principles of DevOps. Every person in the room at DevOpsDays Cuba was focused and listening to what we had to say. This excited me as a speaker and everyone else in attendance. We could see people who really did want to change and transform the cultures within their technology teams and overall businesses to start adopting these modern and proven practices.

    The journey ahead is a long one. The infrastructure challenges remain in place and there is much to be learned, but it was great to see such positive energy and eagerness to adapt. I want to thank everyone who attended DevOpsDays Cuba for making it such an amazing experience. An even bigger thanks to the organizers Enrique Carbonell, Rudy Gevaert, Yadier Perdemo, Manuel Oliver, Mike Rosado, Manuel Morejon and anyone I’ve missed. The event couldn’t have been better and it was a genuine pleasure to visit Cuba, I hope to return soon and reconnect with as many of you as possible!

    About the Author / Conor Delanbanque

    conordelanbanque

    Conor Delanbanque has been recruiting and leading recruitment teams in the DevOps space for the past few years. As well as being exclusively responsible for growing some of the most innovative DevOps & SRE teams in the US and Europe as an external consultant and recruiting partner, Conor also founded the Future of DevOps Thought Leaders Debate, sponsors DockerNYC Meetup on a regular basis and co-hosts ServerlessNYC. You can tweet him at @ConorDevOps, find him on LinkedIn or at events, expos and writing articles on devopsy.kinsta.cloud.

  • Bridging the Gap Between ITSM and Software Engineering

    Bridging the Gap Between ITSM and Software Engineering

     

    Considering the individual roles of IT service management (ITSM) and software engineers, it seems obvious they should work closely together. ITSM tracks internal and external customer issues, and engineering fixes code-related issues (such as defects) that IT cannot fix on its own.

    However, more often than not, coordination between these two departments is poor, which—critically—undermines customer response times and damages application satisfaction. It’s important to remember that the quality of customer response isn’t just about speed, it’s about giving customers insight into the state of the issue.

    The disconnect between ITSM and engineers—caused by individual priorities and semantic differences—makes this real-time insight difficult. Both teams have individual and shared concerns, and separating these while simultaneously improving collaboration is a big challenge.

    Addressing this relationship is an important step in optimizing ITSM and DevOps integration, alongside focusing on IT help desk enhancement requests to product management, and tracking code from build to release and back.

    The Flow of Information on Things that Matter

    Most of the tickets generated by the IT help desk can be handled internally with no help from engineering, but it’s the tickets that do need help that require consideration. For these issues, there needs to be an open flow of information on the things that matter between these two teams.

    IT help desk must give engineering enough information to understand the defect and its impact on the customer. Meanwhile, engineering needs to be able to give the help desk a sense of when to expect the fix, when the fix is done and be able to request more information when required.

    Information that should be shared from the help desk to engineering includes the following:

    • Severity – Engineering needs a sense of the importance of the defect. If everything comes in as “High Priority,” nothing is high priority. The help desk must understand that engineering already has its own internally generated set of defects, as well as defects found by QA and other sources. For these customer defects to be prioritized and fixed in a timely manner, engineering needs to know how impactful they are to the customer.
    • Description – This is perhaps the most important piece of information the help desk can pass along. The more contextual information that can be shared, the better. It’s immensely important to provide engineering with the necessary steps to reproduce the specific defect, which version of your tool the customer is using, the general environment, etc. Sometimes this is best in structured fields, whereas other times it’s important to simply write out everything.
    • Screenshots and other visual information – Along with the written description of the problem, engineering often requires logs and/or screenshots. As the saying goes, “A picture is worth a thousand words.”

    Now that engineering has the vital information it needs to fix the defect, it has a responsibility to provide useful information back to the help desk:

    • Planned release – This should not be a firm commitment, but rather an estimate to give the help desk general timing. While the actual date of the fix may vary, it’s still useful to be able to tell a customer,”This is coming soon” vs. “We realize this is a problem and it’s on our backlog to fix.”
    • Engineering status – While this is a single defect to your customer and to them only has one status, there are really two statuses your organization will care about:
      • Status of the ticket in your ITSM tool
      • Status of the defect in your defect tracking/agile planning tool

    It’s very important that the ITSM tool contains both statuses to provide more information and context to the help desk. Imagine the benefit of knowing that the defect is “In Triage,” “In Progress,” or “In Verification” and being able to provide that information to a customer.

    • Actual release/released – As opposed to the planned release information, this lets the help desk know that the customer’s issue has been resolved. Imagine running a report from your ITSM tool indicating the defects fixed in your most recent release. Your IT help desk can proactively reach out to customers to let them know to upgrade, enabling you to offer more proactive customer communications.

    Finally, and perhaps most important, you allow your teams to talk about the “same thing” even when it exists in different tools and they speak a different language. There are four core ways to do this:

    • Cross tool tracking – Each artifact in each system should have the ID (and ideally URL) of its twin artifact in the other system. If a help desk representative needs to talk to an engineer about a defect, the rep needs to let engineering know which defect it is in the engineering tool, not just in the ITSM tool.
    • Routing – IT help desks often support multiple products, each of which have multiple components that may belong to a different engineering team. The help desk needs some indication which component and/or team should work on the defect. This information will help the IT help desk know which team to reach out to for help.
    • Filtering – Not every ticket in the ITSM tool needs engineering help (that’s a good thing). However, there should be some indication on the IT ticket if it has been sent to engineering. You do not want to send the same defect to engineering twice (confusing) or to assume it’s been sent when it hasn’t (poor time to resolution or issue overlooked).
    • Communication – IT and engineering need a shared way to talk about a specific defect that they both care about. The closer this communication can be to the native commenting that already occurs, the better. Ideally this is not via email as that communication vehicle does not stay attached to the artifacts in question, meaning information can be hard to find or get lost.

    The Benefits

    Customers will love it – Improved communication means happier customers. Customers can plan their upgrades with more certainty, and their trust grows since your organization is able to deliver on its promises. No longer will they feel like their issues are being thrown into a black hole.

    Better engineering focus – Engineering gets pulled in a variety of directions, contending with feature work as well as internally generated defects. The more information it has, the easier it is to prioritize and deliver more value to customer. Without the ability to share the severity of a defect, you risk two undesirable outcomes:

    • The defect is fixed instead of more important work, which hurts all your other customers.
    • Other less important work is prioritized, which can threaten your relationship with this customer.

    Better trust across your organization – Open communication fosters trust and understanding. Providing insight into prioritization and status let’s your help desk feel part of the development process. No longer will staff feel like they’re sending problems into a black hole with no idea when they will be fixed. Enhanced cross-team communication also helps engineering to avoid feeling like work is simply being thrown over the fence. Obtaining more context around a defect is invaluable.

    Next Steps for Total DevOps Integration

    To fully integrate your ITSM and engineering teams, there are still (at least) two other processes to implement:

    • A method for the IT help desk to provide enhancement requests to your product team: These enhancement requests are similar, but distinctly different from the defects described above. Consequently, they need to be prioritized and triaged by a different team.
    • The other process to implement is the traceability from task to code to build and back again. In order to properly inform customers that their problem is fixed in this version of your product, it’s important to correlate features, stories and bug fixes with the actual version of your product that includes them for the first time.

    Bridging the gap between ITSM and engineering teams brings your company closer to customers. While making your organization work as a unified whole is an incredibly complex problem, addressing this relatively small—but vital—piece of the puzzle helps you on your journey to integrating your whole software value stream. This change of mindset—combined with connecting all teams, tools, disciplines and processes in an enterprises’ software value stream— optimizes collaboration, productivity, efficiency and, most importantly, value delivered to the customer.

    About the Author / Trevor Bruner

    Trevor Bruner is a Product Manager for Tasktop Technologies. His background includes experience in financial services, oil and gas and as a Navy submarine officer. Connect with him on LinkedIn and Twitter.

  • Swarms: Adapting Guilds To Scale Agility

    Swarms: Adapting Guilds To Scale Agility

    In my first article in this series, “Using Guilds to Combat Information Silos,” I reviewed the adoption of guilds at xMatters. As our company has grown, we’ve used guilds to keep critical information from languishing in silos—as often happens when an organization scales. Guilds have helped our engineers apply their creativity and critical problem-solving skills to other channels and contribute to areas outside their normal scope. It’s a win-win situation for everyone who participates—and especially for the company.

    Implementing guilds began as a grand experiment. We quickly learned a lot about what enhances or hinders a guild’s success, making adjustments and adding a necessary dash of protocol along the way. But, as is the case with any growing organization, new needs and pain points challenged us to adapt and tackle them. One attempt to solve a critical problem led to an entirely new team model: a body we call the Swarm.

    The Swarm arose as a result of key changes in our release protocol. We used to be very focused on getting new features out the door as fast as we could. As it turned out, many of our customers prefer a more predictable schedule. So we gave our customers  flexible options by maintaining continuous delivery of code to production, with flags to stage larger features or more visible changes. Now, customers who wanted more predictable updates could opt for a quarterly release model while others could continue to access features as they became available and were being iterated on.

    While the quarterly release model improved our rhythm for customers, it also created stricter deadlines. A project that just missed a quarterly release could get delayed by almost another full quarter before our customers could start using it. Throwing more engineers at a problem doesn’t necessarily solve it faster—but, being the engineers we are, we thought about ways to parallelize the problem.

    We knew that guilds were very good at collaborating around a shared problem, so we adopted temporary guilds, which “swarmed” around a shared problem, completed work at an accelerated pace and disbanded. Centered around a specific initiative with a definitive start and end, multiple teams from Dev to Ops collaborated to complete a defined project. For instance, we might form swarms to create new product features, and their lifetime would span a finite number of sprints across multiple teams until completion. Due to its nature of focusing on new product features, swarms differ fundamentally from Guilds in that they are typically not self-forming; each one is an entity intentionally designed for a specific purpose.

    We began our swarms experiment with a large architectural change to roll out a new format for our customer hostnames, which involved six Engineering and Ops teams. We realized immediately that this would be an incredibly challenging undertaking on our tight deadline. A slew of teams was working on different stacks in different programming languages, so there were many competing visions and different contextual understandings of the end goal. It became very apparent that the swarm could not be left to operate in a completely flat hierarchy, as a guild usually does. We needed a leader to help ensure there was no duplication of work, things weren’t falling through the cracks and necessary dependencies were completed in the scrum storywriting between teams.

    Having someone take care of all the logistical problems is one thing, but we realized that in-person communication between all the teams in the swarm is absolutely necessary. We tried staying in the loop with each other by defining a process of posting updates from each team in our swarm’s Slack channel, but this was not enough to keep the project on track. Mattius, one of our very talented operations leads, initiated regular in-person meetings across all the teams, which led to far greater clarity. It was absolutely crucial to have all of the teams together and face to face on a regular basis.

    In the end, swarming on this massive endeavor allowed us to get the project delivered a week before the deadline. It was a challenging and imperfect first run, but it absolutely paid off.

    I leave you with these three recommendations:

    1. Create each swarm with a clear purpose and deadline;
    2. Have a leader to enforce consistent communication; and
    3. Have regular face-to-face interactions.

    The lessons we learned were invaluable for refining (and continuing to refine) our approach to tackling projects for faster release, and we’re continually working to improve and enhance this model. Good luck with all your future swarms.

    About the Author / Nick Fletcher

    Nick Fletcher, Engineering Manager at xMatters, started designing and developing websites back in 1998, when the World Wide Web was the Next Big Thing. In the time since that first website, he’s mastered all the aspects involved in creating great products: design, development and strong leadership. Connect with him on LinkedIn.