devopsy.kinsta.cloud

Tag: devops culture

  • Developers Continue to Prefer Remote Work

    Developers Continue to Prefer Remote Work

    A survey found 83% of developers are working remotely more than half of the time, with more than three-quarters (76%) reporting they have maintained or increased the frequency of remote work in the last year. In fact, three in five respondents (62%) work remotely at least 90% of the time.

    The survey, published by Netlify, a provider of a platform for building the front end of Web applications using the Jamstack framework, polled 6,544 developers in the Jamstack community and found more than half (55%) would quit their jobs if forced to return to an office.

    It’s not clear how much remote work has become a de facto requirement for organizations to attract and retain developer talent, but it’s clear that, in the wake of the COVID-19 pandemic, the dynamic between developers and the organizations that employ them has completely changed.

    Netlify CEO Mathias Biilmann said regardless of industry, it’s apparent that DevOps teams are going to continue to need to make adjustments to foster collaboration across distributed teams. Senior managers will find it increasingly challenging to manage workflows across a workforce that is also witnessing a lot of turnover, he noted. A third of survey respondents (33%) reported having changed jobs in the last year. It’s not clear how much a lack of interaction in the office might be impacting innovation, but at a time when demand for developer talent far exceeds the available supply, organizations will need to adjust to what appears to have become a permanent new reality.

    At the same time, the Netlify survey noted the pace at which emerging technologies are being embraced continues to rapidly evolve—perhaps because of rather than in spite of remote work. For example, the survey found 70% of developers reported using a serverless computing framework in the last year. Nearly half (47%) are also experimenting with edge dynamic sites, the survey found. It’s clear that company culture, at least in terms of technology adoption, plays a large role in attracting developer talent.

    Overall, 96% of survey respondents noted that JavaScript is their primary choice of programming language, with 21% of developers now specifically using TypeScript to build applications. The React framework for building applications, meanwhile, is being used by 71% of developers, with 47% reporting they employed the Next.js version of that framework. Other React frameworks gaining traction include Vite (32%), Sveltekit (15%), Astro (11%), Remix (10%) and SolidJS (6%).

    However, emerging Web3 technologies have yet to gain traction. While 7% to 10% of developers have tried out technologies such as cryptocurrencies and NFTs, only 3% reported using these technologies regularly. A total of 42% of developers either didn’t know what Web3 was or didn’t care about it, while 31% felt negatively about it, the survey found.

    Ultimately, the primary reason organizations invest in DevOps is to increase developer productivity. The challenge is that, as developers embrace more tools and platforms for building applications, the process of deploying those applications becomes more challenging to manage effectively. One way to achieve that goal is to separate frontend and backend software development processes to enable developers to focus more on writing code rather than managing the arcane processes required to deploy it.

  • Everything is DevOps

    Everything is DevOps

    It’s funny how we in IT tend to be absolutists, shoving everything into the current bucket. XML was going to eliminate programming—even though it wasn’t even particularly good at data representation. NoCode was going to eliminate programming (yeah, that’s a theme, I’ll skip the other 50 or so ‘going to end programming’ cycles we’ve been through), AI will eliminate IT, you must have a mobile app, even if no one will ever use it, my OS/browser/language is better than yours and will win—you get the idea.

    We’re currently coming to the end of the “Everything is DevOps” phase. DevOps is a great methodology that helped solve several intractable IT problems but, like everything, marketing and a ton of smart people that want to be “right” have taken it too far. Or have they?

    The thing with DevOps is that contrary to what those who try to make a separate market out of DevSecOps or SRE try to tell you, it was designed to literally encompass all of human IT activity. We can excuse the DevSecOps crowd, since DevOps’ roots had a definite “Security is an unimportant bottleneck” feel to it. The SRE crowd, on the other hand, appears to be another case of “my silo is special”. Or, better yet, a stompy marketer saying “We’re late to the DevOps game! Let’s invent something just like it, claim we do it more broadly and give it our own name!”

    Let’s look at what each does, for those of you who are skeptical. 1) break down silos. Check. 2) Monitor everything. Check. 3) Use monitoring results to improve product performance and stability. Check. 4) Design and implement products and product features. Check. There’s more, but you get the idea.

    Do you know what all of this ignores? You.

    One of the issues that our absolutism brings about is that marketers get to dictate what is important. Sometimes that is what enterprise IT needs. Far too often it is what the marketer is paid to dictate.

    I don’t care if you call it DevOps or SRE, I care that your organization feels like it is getting its money’s worth out of IT—something, I might add, that most organizations did not feel prior to DevOps. At least most of the ones I dealt with. I even had a job where they created a whole new department and hired me to run it to “get around those people in IT.” I spent as much time educating business leaders about not duplicating entire groups like storage services and InfoSec as I did building and deploying new products. So avoiding that kind of issue is near and dear to my heart. And we’re more ‘there’ than we have ever been.

    I do care about whether you call it DevSecOps or not. It should not be different. Shifting security left is the appropriate solution to the double-edged sword of security staffing shortages and increased adversary capabilities. So if your developers are getting secure coding feedback in their IDE or each time a build is kicked off, great! But call it DevOps. Say it is part of the DevOps process. Same with configuration reviews like cloud configuration management—it’s just part of the DevOps process, not a security step to be thought of differently.

    I could go on, but hopefully I’ve spurred your imagination a bit and don’t need to.

    You rock. Streamline your thinking to streamline your processes, ignore absolutism and call it macaroni for all it matters. Just keep doing what needs doing—at the most convenient place to do it. And keep developing, deploying and monitoring the systems that keep the company running.

     

  • Leverage Empirical Data to Avoid DevOps Burnout

    Leverage Empirical Data to Avoid DevOps Burnout

    Burnout is possible in any industry. But it seems to be especially rampant in tech, a space bursting at the seams with inflated expectations and ever-accelerating trends. Nearly 60% of tech workers said they were currently feeling burned out at their workplace, found a study by TeamBlind.

    Limiting burnout and supporting employee mental health is essential, not only for the workers’ sake but for the business as well. During this period, which commentators call the Great Resignation, if employees don’t feel respected, they will likely seek new opportunities elsewhere. Ensuring the needs of engineers are met is a vital target amid an ongoing cloud skills drought, a talent gap to which the DevOps space is certainly not immune.

    I recently met with Eric Mader, senior solutions architect, AHEAD, to understand the root causes of burnout as it relates to DevOps. According to Mader, burnout can be caused by many factors, including work-life imbalances and conflicting egos in the workplace. But above else, burnout is most often caused by management setting unrealistic targets that aren’t based on empirical team performance data.

    What Factors Contribute to DevOps Burnout?

    So, what can cause burnout in DevOps? First, the workplace culture can significantly contribute to burnout. In most remote working models, explained Mader, it isn’t easy to ascertain the correct work-life balance. For example, there may be an unwritten expectation to immediately respond to Slack messages or quickly get back to emails after-hours or over the weekends.

    An “always on” attitude can lead to unhealthy work habits. And while this isn’t necessarily unique to DevOps, it is sometimes unavoidable due to the high connectivity required to perform the job. Another potential contributor to burnout is the team dynamic, noted Mader. “Every team seems to be unique in its own way,” he said.

    Although Agile and DevOps involve very technical work, at times, ego can get in the way of a healthy team dynamic. For example, in many tech teams, there is often a “guru,” such as a senior developer or tech lead, that everyone goes to for advice, Mader said. This person might end up doing most of the work (both technical and interpersonal) and usually feels the most burned out. Or, other wanna-be leaders might take on an unnecessary command-and-control role.

    Personality differences with coworkers could contribute to burnout, but other factors like dealing with toil or managing legacy technical debt can be just as tiresome.

    ‘Move Fast And Break Things’

    According to Mader, perhaps the biggest contributor to burnout is the insistence from executives that tech workers consistently outperform their previous output—especially when past team performance data does not support these goals. Setting achievable goals is great, but when deliverables are out of scope, it’s a recipe for burnout.

    Part of the pressure to increase velocity is the constant illusion that “faster and more is better,” said Mader. Value is often attributed to the number of items achieved in a sprint instead of the quality of each point. For example, a team might deliver 200 story points of work in a sprint, but only 30 of these points carry a significant amount of value. Or, a leader might see a team accomplish 60 points of work in a sprint and then encourage 200 in the next one.

    These sorts of high expectations suffer from basic capacity problems. And the added stress quickly realizes itself as a burden, contributing to burnout, explained Mader. He encourages technical leads not to force more work but instead, “focus on the highest-value work based on the empirical data you have.”

    Five Tips To Avoid Burnout

    Mader shared some tips on how to avoid burnout in DevOps teams. Here are some simple yet effective ideas to maintain happier engineers and a more productive, healthier team relationship.

    Snuff out the always-on attitude. Organizations need to walk the talk when it comes to mental health awareness. Therefore, ensure employees know that their life boundaries are respected. Encourage this by honoring time off for holidays, paternity/maternity leave or personal vacation. Discourage emailing on the weekends and encourage turning on ‘Do Not Disturb’ alerts on team chats and work email accounts.

    Prioritize what gets automated. There’s a misnomer that any automation is good automation. In fact, over-prescription of software development automation can lead to some negative consequences. “I’ve seen many DevOps teams that focus so much time on automating low-impact business processes,” said Mader. Instead, prioritize automation projects that affect more people and those that would be continually used.

    Try out the ‘Fist of Five.’ Ensuring everyone’s voice is heard helps avoid a toxic working environment. One tactic to quickly gauge reaction in team meetings is to use what Mader calls The Fist of Five: One if an idea is totally off base and five if an idea makes total sense. This simple practice could ensure everyone gets to contribute input on new projects.

    Take the quietest person in the room aside. Only the squeaky wheel gets the oil, Mader noted. With that in mind, take aside the quietest person post-meetings to gather their input—you might be surprised. Junior folks who may be less apt to speak their minds typically have a less-institutionalized mindset and could suggest more innovative ideas.

    Use empirical data to inform future goalposts. Finally, Mader encouraged DevOps teams to “use empirical evidence to back up on where you’re going in the future.” Specifically, in the DevOps sphere, empirical data takes the form of metrics like mean-time-to-recovery, or SLAs, SLIs and SLOs.

    Remember that even if a team did accomplish a certain number of goals in a sprint, it doesn’t mean they can produce the exact same results the next time around—they could have been pushed to their limit to produce past results. Thus, managers should aim to move the needle on metrics progressively, not in giant strides that upset morale.

    Take an Opinionated Approach to DevOps

    Nowadays, DevOps teams are spoiled for choice—there are countless tools on the market and an endless stream of best practices and methods. Even within a single company, the DevOps toolset may vary considerably from team to team. To help make sense of it all, Mader encouraged DevOps leads to simplify the transformation journey by setting guardrails and adopting a more opinionated approach around what tools and processes you take on.

  • Why Your DevSecOps Initiative Will Fail

    Why Your DevSecOps Initiative Will Fail

    When you’ve been in and around the security industry for long enough, you get used to the industry hype machine turning a cool innovation into, uh, meh. This hype cycle starts at the RSA conference each year, and folks like me look for new hot stuff on the show floor. For perhaps only the second time in 25 years, I wasn’t at the show in person, but it seems that DevSecOps was the preferred bandwagon on which every vendor jumped. Sigh.

    I’m not sure whether it’s my increasing age or this business that has made me cynical, but here we are.

    To be clear, from a customer’s perspective, there is great value in the concept of DevSecOps, as evidenced by the upcoming DevOps Connect: DevSecOps @ RSAC 2022 virtual event. Having these (sometimes) mortal enemies working together toward a common goal is great, in theory. But alas, in practice (and similar to what happened when DevOps hit the scene), it’s easier said (or displayed in a vendor booth) than done.

    Thus, I’ve come to a sobering conclusion: Your DevSecOps initiative will fail.

    Most organizations will talk a good game for a while and they may even try some stuff, buy some kit, send folks to training and the like. But in the end, despite all these efforts, you guessed it—they will still fail.

    Is this a foregone conclusion for every organization? Well, no. But this isn’t my first rodeo, and I’m pretty sure this one will end like the others—in failure.

    But let’s say you want it to be different this time. Really, you do. And you are willing to take some actions to give yourself a chance at success. If that’s the case, here are a couple of tips to increase your chances, however small.

    Culture

    First and foremost, DevSecOps is not a thing. It’s not a set of tools (no matter what the vendors tell you); you don’t buy it on a VAR line card. DevSecOps (like DevOps) is a cultural shift. Your teams need to work together. They need to collaborate. They need to have shared incentives. And they need to experience the consequences of not playing nicely in the sandbox.

    Cultural shifts are hard. People don’t like to change. They resist. They undermine. They sabotage. It all sounds nefarious, but they do. So unless you can change your culture, you’ve got no shot.

    Shift Left

    Another one of these security-marketing-terms-run-wild is shift left. It means you want to integrate security testing directly into the development process as early as possible. You should be running security checks in the developer’s IDE to ensure they don’t violate security policy (like hardcoding API keys in their source, for example) before they even commit the code.

    You must also integrate security testing into the CI/CD pipeline(s). Thus, you need tooling that works with your preferred pipeline.

    Security Champions

    Security testing will create angst for your developers. They will have false positives. They will have to fix things that they probably don’t understand. It will cause them to miss delivery dates. Of course, they should build it right the first time, but in their mind, they’ll see it as the security folks causing delays.

    The developers will be annoyed, and you’ve got to manage that annoyance—or have senior management quash any rebellions, but that’s not a long-term solution.

    On the other hand, you could establish a security champions program, which anoints (and trains) developers on each team to “represent” security within the development team. They help educate the developers on the tools and secure coding. They diffuse irritation when something goes wrong.

    A strong security champions program is one of the leading indicators of DevSecOps success.

    Break the Build

    Finally, the security team must have the juice to break the build if a critical issue is found during the deployment. We talk about this with many organizations and they nod their heads. But they are lying to me and to themselves. When it comes down to it, someone on the dev team calls someone in senior management and, all of a sudden, that critical issue is now an alert, not a failed deployment.

    Not being able to stop a build will kill your DevSecOps program. It’s like when a quality control issue forces someone to stop the assembly line. If you don’t stop the line, faulty products can result. In a security context, that means some critical vulnerabilities will be much more expensive to fix later.

    So, there you have it. If you can evolve your culture, integrate security testing into your toolchain, empower security champions and empower your team to stop a deployment, you’ve got a chance at a successful DevSecOps program.

    Unfortunately, you probably can’t do these things, and you’ll likely fail. In that case, I have some lovely property by a waterfall I’d like to sell you.

  • Let Me Reiterate – Don’t Rush to Iterate

    Let Me Reiterate – Don’t Rush to Iterate

    We in the DevOps world spend an inordinate amount of time talking about tools and roles—and there are always new tools and modifications of roles to talk about.

    Do you know what we don’t discuss as an industry? The day-to-day grind of being in DevOps. Oh, you can go to forums like Reddit and have those discussions with other practitioners, but pundits don’t discuss what it’s like to actually do the work. I secretly suspect this is because most of us pundits no longer develop code in general, let alone have a working DevOps environment. I certainly don’t code as much as I used to, and I develop more than many of my peers in punditry. For the record, we do have a working DevOps environment (two, but one almost never gets spun up) at my company, but I’ll be nice and won’t get sidetracked by talking about it here.

    We’ve talked about it tangentially, but I wanted to be direct with you all. The vast majority of your organizations do not need to iterate non-stop. And the mentality that you cannot iterate too much is simply false. While the guard gates and German Shepherds that protected production from rogue developers in the past were overreactions, there should be a happy medium. Every time change is introduced in production, there is some amount of risk. And we have a ton of tools to limit that risk these days—feature flags to turn new functionality on/off, auto-run testing before the move to production, etc.—but there is still measurable risk. And “we can just roll it back” only works in small, unsegmented projects. Once you get into nested dependencies (microservices are a great example), then versioning starts to make rollbacks an issue. Knowing the interconnected relationships between what is being changed is a big deal. So it is not as simple as the ::shrug:: “Just rebuild and deploy without your changes,” crowd would have you believe.

    And to a great extent, that is why this pundit doesn’t come out and blather about iteration speeds—the decision to iterate once every six months or every time Bob checks in code is very much an organizational/application one. How fast can the organization handle a full iteration, and how often does the application require it? The answer is rarely the insane number that vendors like to throw out to show you how great they are. There are a whole lot of organization/application combinations whose rate of change is slow enough that they could honestly schedule full-on iterations and plan updates around vulnerability patches in the underlying composite pieces. The majority only need to iterate over months. Very (and I stress very) few need to iterate even daily, let alone more than a hundred times a day. And they are indeed special snowflakes with super-tweaked environments.

    It is kind of stunning to me that a portion of the rise of Agile and DevOps was to free up IT time because we were overloaded. And then once that was done, we decided to crank up the rate of change so that IT was … overloaded. A release takes DevOps time, DevOps time and security time. It takes resources to build, test and deploy. Don’t overdo it.

    Build when you need to. Deploy when it is a benefit to customers. Keep rocking it. We’re out here using your apps, or your apps are grinding behind the scenes, keeping the business running. Either way, you know better than anyone what your app and your org need. Don’t follow the “10 billion deploys a day!” crowd; push for what works. And you are the SME in that department.

  • How to Manage and Engage Remote Engineering Teams

    How to Manage and Engage Remote Engineering Teams

    As engineering teams look to remain remote or hybrid, executives need to fundamentally change how these teams are managed. After tackling the tactical aspects of a distributed workforce—like home office equipment, a comfortable space where employees can get work done and reliable meeting, calendar and communication tools—more nuanced issues come into play.

    Before the COVID-19 pandemic, when a minority of engineering team members worked remotely, those team members knew that they needed to make a concerted effort to connect and to be part of the group. When the majority are remote, however, the expectations of offsite employees have shifted dramatically. For organizations that had built their business with an in-person team, this brings up entirely new questions of communication, culture, loyalty, problem-solving and retention to the forefront.

    Fundamentally Shifting Engineering Team Structure

    Inspiring change requires fundamental shifts in the ways engineering teams work. Our own engineering teams have experimented with some changes that have proven effective, including:

    • A globally revised team structure that embraces a culture of autonomy and breaks larger engineering teams into smaller groups. Smaller groups help engineers feel more engaged, valued and empowered; it keeps meetings short and focused and allows for more parallel streams of work.
    • Overhauling our design process to get a champion or ‘anchor’ from the dev team involved in key features early on. This anchor is responsible for translating product and design requests into technical specs and is the de facto technical lead of the feature. Having this role rotate between team members for each feature means each member gets a chance to grow their skills as a technical lead.
    • Assigning dedicated designers to smaller groups to empower them to make small adjustments mid-flight. This would allow standups to operate as mini working sessions to iron out technical details that would normally be resolved through ‘shoulder taps’ or in-person quick questions.
    • Placing a strong emphasis on sprint retros as a way to make real changes to the way work gets done. When used in conjunction with an initiative to empower devs, sprint retros became a powerful tool to get dev feedback on improving the process.
    • Keeping teams accountable by encouraging transparency and clear communication as a tradeoff for their autonomy. We encouraged note-taking and regular updates, as well as clear goal-setting from the teams at the beginning of each sprint. By holding teams accountable to the goals that they set, it became easy to see when things were and were not working.
    • Experimenting with a four-day workweek to see if we can drive both greater happiness and productivity. This was perhaps the boldest of our experiments; we demonstrated trust in our team and set clear expectations around outcomes rather than hours. This has motivated our team and raised their morale in crucial ways at a time when switching jobs was both tempting and easy.

    Hacks That Worked in Person

    What are some of the things that worked in the office? This list will be different for each organization; for us, it was group brainstorms, in-person one-on-ones, summer Fridays and ongoing vibe checks. Here are some of the ways we hacked these to make them work for remote teams:

    • Group brainstorms: Digital whiteboards like Miro or Mural have been great. Digital sticky notes, too. This has enabled distributed engineering teams to contribute either all at once or asynchronously.
    • Summer Fridays: Give people the afternoon off even if it isn’t technically “leaving the office early.” Or try having everyone shut off notifications on Friday (or Monday morning) so they can get some undisturbed work done before shutting off for the weekend. Monthly mental health days off for the whole company can be a great idea too.
    • In-person one-on-ones: With Zoom, something is definitely lacking. Try spending half the one-on-one meeting chatting casually about things other than work.
    • Vibe check: Open office hours dedicated to work culture; surveys from time to time to gather quick informal feedback or think about other ways to elicit ongoing feedback from every employee, including engineering teams.

    Generate Culture and Well-Being

    What keeps most employees at companies is social bonding, inspiring projects and the possibility for growth. A question a lot of executives at previously onsite organizations are asking themselves is, “How do you fulfill the need for social bonding when most employees, including your engineering team, are remote?” In other words, how do you bond socially and do so remotely? Two goals we have identified are getting team members talking outside of work while avoiding Zoom fatigue issues and ongoing opportunities to share personal successes with colleagues.

    Some of the ways we have built culture are:

    • Having (non-mandatory) Friday lunch and games like trivia, Jeopardy!, games or online games
    • Different ongoing events like photography challenges, health challenges and even an at-home Iron Chef-style cookoffs.
    • Having leaders share how they spent their weekends and time off. We believe in demonstrating a healthy work-life balance from the top down and encouraging our team to take the time they need to unplug and recharge.
    • Figuring out how (and how not) to surface urgent issues via Slack; working on a system to avoid pinging people after hours for non-urgent issues.

    Above All, Manage With Empathy

    These last two years have been exhausting for most people. Hands down, the best way to manage and engage the people you work with—no matter if they are in engineering or marketing—is to be empathetic to the difficulties that they may be going through outside of work. Executive teams can make all the tactical changes to their teams they want, but without managers that really care about their teams, you will inevitably fail to keep the best people around. Empathy-driven leaders lift their team members up and drive the kind of culture people want to be part of.

  • When DevOps-as-a-Service (DaaS) Meets Security

    When DevOps-as-a-Service (DaaS) Meets Security

    One of the most recent IT methodologies to be offered as a service is DevOps, a cultural and practical approach that brings development and operations teams together under one umbrella of work. DevOps-as-a-Service (DaaS) ensures that everything related to the selection, management and maintenance of DevOps tools and infrastructure, including all policies and procedures, are addressed through a centralized team who are able to provide this service to development teams across the organization. 

    From a 30,000-foot view, DaaS represents DevOps capabilities made accessible to practitioners through portals and APIs. Because DevOps capabilities vary among organizations, there is no standardization of these services. For the business, DaaS enhances competition. Application users and developers find DaaS improves overall software development, while specialized DaaS teams say it improves the management of DevOps capabilities.

    What Are the Other Benefits of DaaS?

    DaaS offers many benefits, especially to development teams who no longer want to spend all their time configuring and deploying the necessary security tools to run DevOps pipelines. This means it’s no longer necessary to understand all the differences between DevOps tools and the underlying infrastructure, as the appropriate tools and technologies are automatically deployed for development teams. As a result, developers are able to focus their efforts on creating innovative software at the speed of business. 

    Further, Daas delivers clear business and operational benefits in the form of a more streamlined process for facilitating DevOps across the enterprise. Daas supports governance while also enabling centralized oversight and visibility into development workloads and productivity. As we know, this shift leads to increased accountability among teams and an overall optimization of business processes. Add on resource efficiency and cost savings through tool selection, and it’s easy to see why DaaS has found a home in the world of software development. 

    But despite the benefits, Daas also presents new security challenges that can’t be fully addressed by traditional management solutions. These approaches are often sluggish, expensive, and/or incredibly complex to manage.

    What About Security and DevSecOps?

    Because the principles of DevSecOps proactively address the risks and threats associated with DevOps processes, it has seen rapid adoption in recent years. By integrating AppSec tools and strategies within a DevOps pipeline, developers are able to find and fix security vulnerabilities earlier within the software life cycle—not after the fact. This ability is what allows developers to generate secure and high quality software from the beginning, while also preventing any deployment delays stemming from last-minute security checks in production. With DevSecOps processes in place, AppSec testing becomes seamless and transparent for developers, who need a frictionless experience to facilitate steady and reliable software development and deployment. 

    DevSecOps is a  natural fit for the as-a-service delivery model. Similar to DevOps, AppSec involves various tools, policies and processes that demand a certain degree of professional expertise—not to mention the time and resources it takes to to select, maintain, and deploy them to drive DevSecOps practices. Security and development teams looking to  implement and manage DevSecOps can reap major benefits from a centrally defined and managed process, to be administered by a specialist team and delivered “as a service” to those in charge of DaaS.

    Standardizing AppSec through a DevSecOps-as-a-Service offering would guarantee security is applied evenly across all DevOps activity, thereby allowing businesses to produce secure and high-quality software consistently while also supporting AppSec compliance demands. This shift also establishes a foundation where a holistic and homogenous view of AppSec can be found, one that is critical to the assessment and management of application and business risk. 

    Developing DaaS Metrics

    First, don’t try to boil the ocean by measuring all applications at the same time. Instead, choose an application that’s supported by DaaS processes and appoint a specialized team to develop metrics for the application. This team can look many ways—but would likely include stakeholders, developers and other DaaS-supporting roles. Service-level indicators (SLIs) and service-level objectives (SLOs) are also baked into this model, which are chosen to best suit the requirements of an individual application. 

    Once the most valued SLOs of DaaS are determined, the service can be tailored to identify the most critical SLIs. With this knowledge, things like instrumentation, telemetry and analysis requirements can be identified and used to design alerts, dashboards, analyzers and more. Next, setting up policies and procedures around accountability establishes a complete solution. And once the solution for the chosen application is tested, confidence kicks in and creates an easier process for adapting and expanding the process around other applications. 

    This means DaaS can provide many key capabilities to both the organization and the teams who rely on them. A specialized DaaS team can then figure out the best metrics for larger implementation on any application. As a result, SLOs can be evolved for other future applications, to be used as a baseline for future improvements. 

     

  • Shedding Light On Toil: Ways Engineers Can Reduce Toil

    Shedding Light On Toil: Ways Engineers Can Reduce Toil

    It comes in many shapes and sizes and is embraced as a necessary evil. It lurks in the shadows, emerging now and again to stealthily creep into our workflows, where it feasts on our perceived shame. If not pruned, it grows and wraps its tendrils around the workforce, draining the energy out of every digital worker across departments. We all have it and most of us believe we suffer alone.

    I’m talking, of course, about toil.

    Dramatics aside, toil is common throughout most programming and DevOps positions, whether we like it or not. And when it comes to platform engineering, the chances of encountering toil is high. You can think of toil as those tedious workarounds that should be automated but aren’t. This could be due to a lack of standard configurations for deployments. Or, perhaps engineers must copy and paste data from one module to the other—an integration that has not yet been automated.

    One of the hallmarks of the SRE role is to spot and reduce such toil—hopefully before it gets out of hand. But, what exactly is toil? And, how can SREs reduce it?

    I recently met with Jake Englund, senior site reliability engineer, Blameless, and Matt Davis, intuition engineer, Blameless, to uncover some tips on spotting and eliminating toil in the daily life of an SRE. Below, we’ll attempt to define toil and explore some techniques to defeat this ghoulish foe.

    First, What is Toil?

    Toil is tedious. Englund defines toil as repetitive manual tasks that are, ideally, automatable. “It’s not gratifying work; it lacks enduring value.” James Curtis similarly described toil as “tedious actions that really have no enduring value.” Toil is similar to technical debt in that, if left unchecked, it can hinder productivity and contribute to burnout.

    Toil is invisible. Toil is the invisible work that goes on behind the scenes, says Davis. For example, at your team meeting or stand-up you might say something like, “I’m deploying this fix.” There’s usually no need to explain the nitty-gritty details of your exact step-by-step process, as it would be too far in the weeds. As a result, manual and repetitive tasks are often not given the spotlight.

    Toil is persistent. Toil might be readily apparent, but it’s not as simple to eliminate; attempts at fixing toil could potentially create new toil. Tedious tasks may also be essential to keeping core infrastructure pieces up-to-date. According to Englund, what often ends up happening is that only the specific engineer knows the niche mechanics of what they’re maintaining, making them irreplaceable—a power that some employees might abuse. “If you’re the only one who knows how to do something and you’re hoarding that information, it leads to inertia against resolving toil,” said Englund.

    Toil is pervasive. Toil is not limited to SRE or engineering. From UX to management or marketing, within most positions, you’ll encounter a degree of toil. This is because the ‘work as prescribed; is often much different than ‘work as done’, said Davis.

    Four Tips To Reduce Toil

    So, what can organizations do to conquer the nightmarish reality that is toil? Especially those that don’t have a dedicated engineering team to reduce drudgery?

    1. Find It!

    To reduce toil, you first have to find it. Sounds simple enough, right? Well, discovering toil might not be as obvious as it sounds. “You get into these toil situations where you might not even know you’re in it. How do you know that toil is encompassing what you’re doing?” asked Davis.

    To find toil, the Blameless team has a weekly session open to the whole company to examine ‘work as done.’ According to Davis, this has been very beneficial for discovering where toil occurs. It also helps garner perspectives from roles throughout the entire company.

    2. Standardize and Automate

    Next, a proven method to reduce toil is standardizing your engineering processes. Standardization, said Davis, is “another word for removing ambiguity.” For example, SREs can benefit from having a common series of steps to follow when responding to alerts. Runbooks that aggregate data to respond to issues can reduce toil in the data collection process while also helping improve mean-time-to-discovery and recovery rates.

    Standardizing change management is another way to reduce toil. If engineers are approaching change management in various ways, this could cause additional review and slow down the release process. Removing ambiguity from the change lifecycle can alleviate toil down the line.

    When you find yourself having to retain mental models for configurations, it’s a good sign that standardization is required, explained Englund. For example, perhaps some configurations are needed in some environments but are commented out in others. Documenting the specific configurations for different development environments (and automatically populating configurations) is one way to reduce toil and the chance for human error.

    3. Proactively Monitor

    More proactive monitoring is another way to reduce toil, according to Englund and Davis. “Responding to a crash loop is responding too late,” added Davis. Instead, he advocated that SREs look toward leading indicators that suggest the potential for failure so that teams can make adjustments well before anything drastic occurs.

    If SLIs like error rate and latency are getting bad, you must take reactive measures to fix them, causing more toil. Instead, proactive monitoring is best to see the cresting wave before the flood. Leading indicators could arise from following things like data queue operations connected to servers or the saturation of a particular resource. “If you can figure out when you’re about to fail, you can be prepared to adapt,” said Davis.

    4. Practice How to Improvise

    One major caveat of standardization is that you’re inevitably going to encounter edge cases that require flexibility. And when an outage or issue does arise, the remediation process is often very unique from case to case. As a result, not all investment into standardization pays out.

    Alternatively, teams that know how to improvise together are proven to be better equipped for unforeseen incidents—one study on Arctic Sea emergencies found collective improvisation to be imperative for emergency response situations. Similarly, collaboration is vital for software teams. Davis calls this the “practice of practice.” If you don’t practice backup restores, for example, teams may flounder in the case of a database outage. Yet, since every remediation effort is unique, it really boils down to learning how to collaborate effectively as a team. “We can gain a lot from getting together to practice,” added Davis.

    All the Small Things

    Toil is like manually typing an email signature every time you are about to hit “Send.” Maybe you’ve just been putting off configuring your signature in your email client. Or, perhaps you like to write a custom sign-off for each recipient. Regardless of the reason, the longer this repetitive motion is required, it’s a stick in the spokes of productivity.

    In the grand scope of things, this tedium can be a drain on time and resources. For every extra keystroke or click, every un-automated workflow or every copied-and-pasted code block, a small degree of toil is piled onto the daily workload of an engineer, dragging them away from focusing on what really matters.

    Standardization, automation, proactive monitoring and collaboration are all helpful elements to consider on your journey toward reducing toil. But of course, there is the trap of increased software development automation—the time saved building automation must outweigh the upfront development cost (and ongoing maintenance effort). Otherwise, you might end up creating even more toil.

  • How to Avoid Crappy Culture and Keep Engineers Happy

    How to Avoid Crappy Culture and Keep Engineers Happy

    Software engineers know how much they are worth—everyone wants them and there aren’t enough of them to fill that demand. While the need for tech talent is greater than ever, the pandemic has shifted priorities for many engineers. And they’re not alone. It’s a reality shared with millions of others in a variety of fields around the world.

    If the so-called Great Resignation has taught us anything, it’s that everyone is tired of crappy workplace culture. People are beginning to question why they’d stick around at a place where they don’t feel valued and supported. And for those companies not investing time and energy into their culture, good luck holding on to software engineers. They know their value!

    In this uber-competitive environment, the best way to keep software engineers happy and make the company a desirable place to join may seem counterintuitive, but hear me out: Just back off. Trust engineers to do their jobs well.

    Put in place the building blocks for a supportive environment where people feel comfortable taking risks and voicing their ideas—even contradictory ones—in positive ways. Then, get out of the way. Leaders have a responsibility to make these values stick and to hire people who fit them, even if that means not always picking the most qualified person available.

    Don’t Be Afraid of “Dumb Ideas”

    I’ll be the first to say it—I have a lot of dumb ideas. I’m thankful to have worked in places that encouraged me to share those ideas. It taught me to focus on building a company culture that values openness to new thought processes. Because you never know when something that might seem like a dumb idea could actually become the next big thing. 

    No one wants to feel shamed for speaking their mind. Encouraging people to share their ideas, even those that aren’t fully developed, fosters a sense of trust in the team and can lead to greater creativity and innovation. It also makes the company stronger by reducing stress on employees and avoiding negative outcomes like burnout and other health concerns that may ultimately lead to turnover.

    Getting to this point of psychological safety in the workplace is not an easy process. It has to be highly intentional given the standards that are often at play in the professional world. For too long, people have been thought of as replaceable cogs in an organizational structure, not as humans with unique and valuable skills, talent and shortcomings.

    It’s about recognizing that people are human and often want to feel authentic in how they work. That’s why it’s valuable to talk about new ideas to improve the company and its products, as well as talking about how you talk; set a defined standard for how important topics are discussed as a team. 

    Fostering a supportive culture doesn’t mean there aren’t disagreements. But there are ways to offer contrary opinions without being a jerk about it. This mutual respect between engineers allows people to look at things with a critical eye without hurting anyone’s feelings.

    At CodeSee, we do a weekly product review. And this often leads to statements like “the product does this, but I wish it did this.” This isn’t a criticism of anyone, and everyone knows that. It’s everyone helping each other in the pursuit of making things better.

    Don’t Hire Assholes

    Instilling these values in a company is a rewarding experience that allows engineers to flourish. But none of that matters if they don’t also show up in the hiring process. The same things we place importance on in our day-to-day operations are top priorities in hiring: Humility, trust and self-reflection.

    We’d rather have a decent engineer that fits our culture and embodies these values than a great engineer that isn’t supportive of others. We don’t subscribe to the idea that it’s OK to treat people like dirt just because you’re a genius.

    These ethos show up in the questions asked to prospective employees. Two of our go-tos are fairly simple. We often ask interviewees to define three strengths and weaknesses. It’s a number that requires some real introspection and helps us see if a person knows where they could stand to improve and have identified personal growth strategies. We’ll also ask, “When we call previous managers, what will they have to say about you?”

    These questions are meant to tease out some self-reflection, naturally. If they can put themselves in the shoes of their past managers, that shows a high level of self-awareness and empathy.

    Surprisingly, these questions have gotten under the skin of more than a few applicants. I’ve actually had people yell at me, the person doing the interview, because they’ve been made to feel uncomfortable. Obviously, that kind of response told us everything we need to know.

    Tools for Success

    Institutional knowledge is an asset to any team. However, it puts additional strain on senior engineers, who are already stretched thin, to hold that information in their head and teach/explain it to less-experienced colleagues. 

    This is something that needs to change to free up senior engineers to focus more on the big picture. We can solve that with tools to visualize code and by investing in models that provide these models to engineers to improve training. 

    Today, many engineers spend 60% of their time reading code. What if we could reduce that to 40%? Or 20% or even 10%?  If all our engineering talent spent 40% of their time doing other stuff, think about where we could be?

    However, productivity tools can be a double-edged sword. Instead of giving more freedom and trust to software engineers, some companies are using technology to wring every last ounce of productivity out of the workforce, efforts that are surely doomed to fail by encouraging talented engineers to start planning their exit.

    The bottom line is that everyone needs engineers. Losing at least some is inevitable. But if you are thoughtful about creating a supportive, innovative culture—and maintaining it—then it’s a lot easier to compete.  

    But it’s no simple task. Every decision you make—from who you hire to the products you build to the way you talk to each other in meetings—plays a role in shaping culture. Engineers will be compensated everywhere they go, so the best chance to get ahead is to build the best, least-crappy culture possible. 

  • The Problem With Security

    The Problem With Security

    As an IT professional with decades of experience at every level of technology and organizations—from cell phone prototyping to banking enterprise architecture, from entry-level to CTO—I can state definitively that I have spent an inordinate amount of time dwelling on and dealing with security. And I know I’m not alone. This is a common theme. You could almost make a sitcom out of it. “What are your developers doing?” Uhhh … writing code? “That’s security!” I could go on, hitting on everything from users to cloud instances and you could fill in the “That’s security!” punchline every time.

    That’s a problem. But not in the way you might think. I spent many years thinking I was somewhat unique in that I always rose to the occasion and approached security issues head-on. Most of the organizations I’ve worked for didn’t have dedicated security (I’ve probably written about that before), so some of us rose to the occasion.  Nowhere near all of IT, but enough to feel like we were protected.

    I now realize that I’m not all that unique; that more of us are doing the “learn security to be secure” thing than are not. I see it in modern organizations all the time. And I think that the real problem is this: Most of these people, like me, did not see themselves as security staff—but they are. Most organizations don’t recognize them as security staff—but they are.

    We’ve long harped on cultivating security recruits from this pool, and I’m now going to suggest that you go even further. Make these people part of your planning. If you know that person X on the ReallyCoolApp team is ferreting out security stuff, ask them to just send an email to update InfoSec once a week. Don’t make it a huge burden, and don’t try to force them into a pure security role if they don’t show the aptitude and interest (you really don’t want unwilling security staff) but make use of their talents and drive. Certainly, if this person is doing a good job of making certain ReallyCoolApp is mostly locked down and DevSecOps tools are running against it for a secondary validation, then your limited InfoSec resources can be directed elsewhere.

    Replicate this throughout the organization, and “That’s Security!” becomes a benefit rather than a hassle to try to lock down. You already do some of this—since the automation of passwords, most identity management is handled by operations staff, with security merely providing policy and being available to ops if needed. I’m simply suggesting that you expand this type of approach to include every IT department (and some non-IT departments in distributed environments). Don’t recruit everyone to be InfoSec, but recruit them to do a little bit of InfoSec in their area of specialty.

    People have always said “Security is everyone’s problem,” which is true, but it’s far too vague to be useful. This is a workable intermediate. “Security is every team’s problem” and recruiting the people on each team most interested in InfoSec to formalize their involvement is a better approach.

    And keep rocking it. We need security everywhere, because ne’er-do-wells have made everywhere a part of the InfoSec battleground. So recruit people everywhere without moving them from their current roles/responsibilities. Don’t let the sweet systems you’ve nursed through the years get whacked by attackers when a little recruiting and nurturing will fill the gap.