devopsy.kinsta.cloud

Category: Enterprise DevOps

  • Development Versus QA: Ending the Battle Once and for All

    Development Versus QA: Ending the Battle Once and for All

    It’s an accepted norm in software engineering that quality assurance (QA) engineers and developers will be perpetually at odds. We are so used to this dynamic that some organizations have decided it’s a good thing. They believe some animosity is beneficial, because QA has to hold developers accountable for good quality, and developers need to pressure QA to not impede time to market with too many tests.

    I disagree. I think infighting between development and QA is a sign of a larger, more endemic problem: They can’t align on who is accountable for quality at what stage, and more importantly, what is mission critical.

    Developer Versus QA Incentives

    Developers are incentivized to ship as much code as possible, as fast as possible. This is because the KPIs they care about most are temporal in nature (time-to-market, feature velocity, sprint velocity, etc.).

    However, when it comes to QA engineers, the things that incentivize them are somewhat counter-intuitive. You might think the most important goal for QA engineers would be to ensure quality, which is at odds with time-to-market and velocity, because more quality assurance will consume more time and resources. QA engineers will tell you (because they believe it) that this is their incentive and the source of the conflict. What nobody’s able to admit is that a QA team’s incentives are actually not to ensure quality but to ensure they are not blamed. They get rewarded by having fewer bugs or other issues that can be blamed on them. 

    The reason why minimizing blame is the number one priority for QA engineers is that in the QA realm, there is a general acceptance that bugs are always going to make it to production, no matter what. This is something we expect because a 100% guaranteed bug-free product would take years to ship rather than weeks, and would therefore be economically unviable. Since they know there will be problems to deal with no matter what they do, they want to show that they did everything in their power to prevent those problems. Naturally, they want to write as many tests as possible to minimize the risk of bugs that they should have caught. But since it’s impossible to write an infinite amount of tests, they have to prioritize what to test for.

    A QA team is given no data by which to prioritize what to test, so this prioritization is essentially a guessing game. It may be an educated guessing game based on experience and expertise, but it’s still predicting what users are most likely to do on an application without objective data as to what they really care about and how they really will use the application. QA teams may add a good amount of structure to their brainstorming and guessing so that they feel like it’s a rigorous process, but without hard data, it never will be.

    The fact that there is not an objective, data-driven method by which testing priorities are set means two things. First, it means potential quibbling between QA engineers and developers over what should and shouldn’t have an end-to-end (E2E) test built around it. Second, there’s likely to be disagreements over test suite size and what that will do to run time. Since the QA engineers’ incentive is to not get blamed when things go wrong, they naturally want to write as many tests as possible. The more tests you write, the higher the chances of bugs being detected by one of them. But as we discussed, developers are incented to maximize speed, and want the product out the door as fast as possible. The more tests that have to be run, the longer it takes the product to get to market.

    Without data, your QA strategy devolves into an argument between developers and QA engineers. Different organizations resolve this argument in different ways. In my experience, I see some teams that don’t have any E2E tests at all. These organizations have much more of a speed-centric, DevOps-first mindset. At the other end of the spectrum, I see companies that have over a thousand E2E tests, and it takes two days to run a regression cycle. The former, where developers have won the argument, accepts a substantial amount of risk when they deploy. The latter, where QA engineers have triumphed, is unwilling to sacrifice the feeling of security that comes with thousands of tests to achieve fast deployments.  

    Testing Is Too Subjective 

    Too often, organizations burn out and break down due to constant infighting between silos; sometimes between QA and development. This is a common management problem that consultants (including myself) often work on. Yet, this battle between QA and developers continues to play out over and over again in software organizations, with seemingly no end in sight. But I don’t believe it’s destined to go on forever. I envision a future in which QA and development live in harmony.

    So, how do we end the eternal battle between development and QA? The way to resolve this conflict is to realign both sides’ incentives and departmental goals. And the way to do that is by basing decisions on objectivity instead of opinion. 

    When arguments over different opinions on testing strategy arise, those opinions are subjective; they are informed by individuals’ own personal incentives and perspectives. An objective way of defining a target or scope eliminates the space for argument. Obviously, two departments working together toward a shared target are going to operate better than two departments that have competing incentives. This alignment requires an objective standard by which we decide what tests are important—what we should be testing and what we shouldn’t.

    A lot of people reading this might be thinking, “Well, we have an objective standard. We have a process that determines our testing priorities.” But let me ask you this: If your QA engineers were swapped out with a group of equally smart and capable professionals, would they come up with the exact same testing strategy and set of tests to prioritize in the testing suite? The answer, of course, is no. It wouldn’t be exactly the same. And if it’s possible for a different person to look at the same data and come to a different conclusion, then the process is by definition not objective. 

    Bringing Objectivity to Testing

    When prioritizing what should be tested, you need to know what will be most important to your user. So instead of having people guess what’s important to users, why not simply let your users tell you? 

    Machine learning has unlocked the ability to bring objectivity to testing. By analyzing real user behavior on your app, you can prioritize and orient your test suite to what your users actually care about. You will objectively know what to test based on actual user behavior, you’ll catch more bugs that actually matter to your users, and you’ll catch them much sooner, with fewer tests than if you’d come up with them yourself. The quality of your applications will greatly improve without having to create unnecessary tests that clog up JIRA boards and deployment pipelines. Both sides win. 

    A New Day for Features Testing

    This also enables a seismic shift in features testing. With new features, we don’t understand what’s important to users yet because we don’t know how they’re going to use them. So how do we test for it? In the near future, machine learning and autonomous testing technologies are going to be able to solve this problem simply by leveraging experience. 

    We label many things “new,” but truly new things rarely enter the world. This is especially true of user interfaces (UIs). UIs should be similar to other UIs. They should be intuitive to brand new users. They must be fairly familiar to customers so that they can use them without having to relearn them. Machine learning and autonomous testing technologies can easily compare new features to feature sets that have been live in other applications before. When the datasets are similar, such as with UIs, login pages, checkout pages, etc., you can reasonably predict from past experience how people are going to use them, and leverage that data to create new tests. The new tests won’t be perfect, but they’ll be a lot more accurate than QA engineers taking their best guesses. 

    One Team, One Mission

    You can imagine this future where your users are constantly informing a machine as to what should be tested in regression, and the amassed experience of thousands of other applications that have similar feature sets is available for analysis. This mass collective experience, which humans couldn’t possibly create, is going to be used to decide what your future test suites should look like. The battle between developers and QA engineers will be over, because the testing strategy will be aligned and there won’t be any argument over what should and shouldn’t be tested. 

    Now, you’ve got a team that has an objective, data-driven balance of speed and testing, with much greater confidence that development can move forward quickly while ensuring quality. Both sides can coexist peacefully and finally work as one team with one mission.

    — Erik Fogg

  • SRE (Part 1): A Modern Overview

    SRE (Part 1): A Modern Overview

    Site Reliability Engineering (SRE) is a topic that over the last several years has become a popular discussion across many companies I have interacted with. This article is the first of two that I’m passionate to share, as I’ve come across many organizations who are trying to build and implement this ideology, while all are asking the same questions—What is SRE? Who are SREs? How do we get it? Where do we start?

    I—just like everyone else—have opinions around this topic. However, there is a common ground between us all. SRE is not only about tooling and tech, it’s not only about the unicorn hires you can make, but it is primarily a cultural shift within companies. It changes the way developers, product owners, admins, general engineers, etc., all interact with one another. You have to shift your operating model and how you approach building in your organization.

    Now, just as a disclaimer these are just my opinions and experiences with building organizations like this and talking to other companies who have implemented or are currently implementing. There is no prescription for building SRE. Each and every company will implement at their own cadence and own internal philosophies. You will, over time, build it in a way that fits your organization and internal operating mode. Do not move too fast and attempt to force the model into your organization. Fail but fail gracefully and learn from those failure. You’ll get to your Narnia at some point.

    Definitions

    Let’s start with some quick definitions of the acronyms we will be using. These will be short explanations but give you the general outline. 

    • SRE: Site Reliability Engineer, an individual who possesses an interest in infrastructure and operations, with a software engineering background.
    • SWE: Software Engineer, someone who writes the application/service software. 
    • Infrastructure SRE: a variation of SRE that works on infrastructure related projects (monitoring, provisioning, IaaS, etc.).
    • Application/Service SRE: a variation of SRE that is dedicated to specific application/service group. These are the groups who are building the products that generate revenue for your company (hopefully).

    The definition of SRE is quite scattered and I don’t think anyone truly understands/knows what it means—and no, unicorn’s will not show up below.

    This is what SRE means to me:

    • SRE is a core group of individuals who have a wide array of skills. They are more of a technology generalist than specialist. These skill sets range from operations, infrastructure, networking, development, hardware, distributed systems, monitoring, stability, capacity planning, software engineering, etc.
    • SREs are responsible for architecture and implementation of technical infrastructure and supporting services, focusing on stability, security and scalability of those platforms.
    • SREs should be building standard best practices, platforms, services, infra, etc., that reach the global organization. 
    • SRE is not just about the technology. SRE is a mindset, thought process and cultural shift. 
    • SRE shouldn’t be made mandatory for everyone in your company. Teams will have the choice to fund SRE support if they need/want it.

    Areas of Responsibility

    Technology has so many different aspects that it may seem hard to narrow down what SREs should actually be spending their time and energy on. Since these individuals may not be directly building public facing products or features (depending on your company, of course), what exactly will they be doing?

    This by no means is limited to the list below, but should be the general areas in which you start to focus your SRE organization. Without these underlying infrastructure plumbing being properly built, monitored and scaled, it will be difficult to preach this new cultural mindset to the outside team members. The items below may seem very infrastructure heavy—and some may be. Many of these can be split between application SRE and infrastructure SRE. One is the primary producer, while the other is the primary consumer.

    • Monitoring.
    • Configuration management and automation.
    • Infrastructure services/networking/platforms/architecture (including hardware and gen compute).
    • Infrastructure tooling and capacity planning.
    • Big data/data warehousing/data analytics. 
    • Documentation and runbooks.
    • Incident response and incident management process.

    Is SRE a Single Team or Multiple Teams? 

    There is no prescription, and results will vary depending on multiple factors. However, I recommend that this is split logically into multiple teams if you have the staff to accommodate. You can start small and have a generalist team that covers multiple areas and eventually break into smaller, more manageable pieces. The areas are:

    • Infrastructure (hypervisor, storage, operating systems, containers, automation, SDN).
    • Observability (monitoring, telemetry, event correlation, trend analysis, IN&IM).
    • Tooling (custom tools, configuration management, developer experience tools).
    • Services (databases, message queues, orchestration, micro-services).
    • Apps (mostly application side knowledge and ability to support/troubleshoot).

    Even though they are split in some sense, the reporting chain must remain centralized throughout the organization. Decentralizing in my opinion will break the effectiveness and communication of the team and result in an unsuccessful SRE implementation. SREs should report to like-minded individuals whose mission statement is to push SRE agenda and philosophies. Agenda is not the only reason. A central hierarchy allows for governance around tooling, best practices and general technical/architectural guidance. From that, you can allow those folks to cross pollinate in other groups and teams and spread those standards across the board. A breakdown in structure and communication will inevitably cause a deterioration in the mission.

    Another reason for centralization is that it allows you to make sure your SRE hires do not become a dumping ground for operational work the app teams do not want to do or have pushed off. I have seen this happen in the past; it discourages the new hires and isolates them. They do not have a mission or a central team to work with and end up building things off on their own that only fit the needs of who they are supporting, instead of seeing the bigger picture.

    The application SWEs should still be doing operational related work, since they are the ones who created the need for it. The application SRE can help offload some of that work and find ways through the central chain on what tools others are using to offload. Operational issues can quickly turn into error budgeting for the application teams.

    Your SREs will likely be split into two disciplines: application SRE and infrastructure SRE. This allows you to cover both sides while pushing the same mission throughout all the teams. Infrastructure SREs are out there, and there are plenty of them. I have seen it come as a natural evolution of DevOps, system engineers and system administrators. You should focus on building your internal talent pool and then going outside for hires.

    Having some of the SWEs on the team volunteer and shift work over to SRE helps speed up the process for that specific application. You can leverage their knowledge of the code base and have them learn and understand the centralized mission and tooling to bring down to their stack. These individuals can also train the central team on the app stack.

    However, not every team in your company will need dedicated SRE support. Part two of this series will be a deeper dive into application and infrastructure SRE, and how to organize your teams from a management perspective.

    — Anthony Caiafa

  • DevOps Chats: Mobile, Monoliths & Microservices, With LightStep

    DevOps Chats: Mobile, Monoliths & Microservices, With LightStep

    Daniel “Spoons” Spoonhower made a bit of a name for himself while he was at Google, as did his two co-founders of their latest company, LightStep. All three were there at the birth of the microservices wave that has swept through the software world.

    They left Google and started LightStep to help all developers build better, more scalable applications.

    With its new release, LightStep has pushed the bar higher. In this DevOps Chats podcast, Spoons tells us what they are doing and where he sees things moving.

    As usual, the streaming audio is immediately below, followed by the transcript of our conversation.

    Transcript

    Alan Shimel: Hey, everyone, it’s Alan Shimel for DevOps.com and Container Journal. You’re listening to another DevOps Chats. I’ve got a great guest to talk with today, I hope you’re going to enjoy it. I’d like to introduce you to Daniel Spoonhower, better known as Spoons. Spoons, welcome to DevOps Chats.

    Daniel Spoonhower: Hi, great to be here.

    Shimel: So, Spoons, you are CTO and Co-founder of LightStep, and though some in our audience are probably familiar with LightStep, being that this is the first time we’ve had someone from LightStep on our podcast, I’m gonna ask you to kinda do some of the heavy lifting here and kinda—

    Spoonhower: Sure.

    Shimel: – tell our audience a little bit about LightStep and what you guys do.

    Spoonhower: Sure—sure, sure. So, LightStep provides simple observability for deep systems. We think observability is more than just metrics, logs, and traces, it’s really about being able to navigate from effect to cause. So, whether you’re pushing into deployment, you’re on call and you got paged or you’re just trying to improve performance, observability tools like LightStep should be able to give you the insights that you need to understand what’s happening in your services and across your system.

    Shimel: Excellent. And just by way of, again, giving the audience the background—Spoons, why don’t you tell them a little bit about your own personal journey? You are a Co-founder and CTO, but there was life before LightStep.

    Spoonhower: True, yeah. So, yeah, I’m Co-founder and CTO and LightStep, but we have three—all three of the Co-founders are technical, and so, we all have done technical things and we all have done a lot of non-technical things, too, which has been pretty fun for me just to learn about finance and about commercial real estate—who knows what.

    Shimel: Yeah.

    Spoonhower: But yeah, I’m an engineer by training. Before this, I was at Google for a while. I worked both on their internal infrastructure team, which was super interesting, as well as on the cloud platform. And I have various other experiences at big and small companies. I was at grad school for a while doing academic stuff, so—yeah, a lot of different things.

    Shimel: Excellent. Alright. So, before we—I know you guys have some new product announcements coming up, but you know, LightStep is, as I mentioned, new to our audience here on the podcast, but it’s been around five years, right? And I think you gave everyone kind of the—you know, the brief elevator pitch about it. But it really, you know, in an age where even just since I’m doing DevOps.com, right, Spoon, we’ve seen APM turn into AIOps and companies go public in this space and everything, several. But yet, you know, the need to kinda monitor, manage, see what’s happening is still here and I don’t wanna say it’s getting easier or—you know?

    Spoonhower:  [Laughter]

    Shimel: Because I think it’s just, you know, natural. Things get more complex as we continue to build and so forth, and I guess that makes the monitoring easier, but when does it get simpler?

    Spoonhower: Well, yeah, so I think the situation has gotten a lot more complex. That might be because of things like microservices or serverless or things like that. I actually think a lot of complexity comes from the fact that we no longer run our whole applications. Like, we use DynamoDB, we use third party fraud detection service, right? There’s a lot of pieces of our software that are not even ours any more, so that adds a lot to the complexity.

    Shimel: Yeah.

    Spoonhower: I think the simplicity has to come from focusing more on the use cases. Like, what are you as a developer actually trying to get done? And then, I mean—I think this is where people kind of get into things like AIOps is, like, using software to sort of harness the data that we have to help you in that task. I think that can be a real-really nice as a developer, right, to not have to understand all of the depths of what’s going on underneath the hood.

    Shimel: Sure. I mean, I just find it somewhat ironic that, you know, we’re gonna break down silos, we’re gonna break up this monolith and go to microservices so that we can deal with smaller, more easily digestible threads, and then we’re gonna orchestrate all of this with a great tool like Kubernetes. And then I look at it, and I’m not the developer you are—I’m not claiming to be, or the technical person—but damn, it gets…it’s just the nature of the universe, at some level.

    But anyway, you guys are well aware of this. It’s acute. This is the world you play in. You’ve got some new product announcements. Let’s talk about ‘em.

    Spoonhower: Yeah. That’s right, yeah. So, I mean, the history of LightStep a little bit is that we started as a tracing tool, collecting data sort of—exactly to address the situation you’re talking about. You’ve broken your application into some pieces and you want to be able to understand the big picture, right, and put them back together.

    And what we’ve done actually is to continue to integrate additional data sources into that, so that we can provide sort of more user-centric things. So, what we’re rolling out right now is a couple different pieces. One is error analysis, so being able to understand, you know, you have a spike in errors in your service, harnessing that data, whether it’s traces or logs or metrics, to understand what the cause of those errors are. We have a detailed rollout analysis, so even if you’re in the middle of a rollout, so you’re saying you’re gonna canary or blue-green or something like that and you wanna understand what’s happening, we’ll do that analysis for you and do a side by side to tell you what’s different about the new release.

    And one thing that’s pretty exciting is that part of this is integrating metrics into the platform. Metrics are a tool that everyone has been using forever and ever and ever, but you know, there’s a lot of metrics out there, and part of what we think is important is that we’re able to focus on the right metrics, the metrics that are actually able to explain what’s happening right now.

    Shimel: Absolutely. So, this new release, April 2nd, is the key day?

    Spoonhower: Yeah.

    Shimel: So, people listening to this can go check it out right now at LightStep.com.

    Spoonhower: That’s right. We’ve got a free developer tier that you can try. We also have a really cool sandbox that’s sort of, it’s a sort of interactive playground where you can check out features and you can basically walk through one of these scenarios. So, say you’re doing a release, say you’re trying to debug an error or something like that. It’s not—you know, it’s not totally constrained, but will kind of show you along and give you a bit of a guide that you can still explore the rest of the product while you’re there, but really trying to help people kind of put two and two together and understand what’s happening.

    Shimel: Yeah, no, I love that, man. Because I gotta tell ya, we’ve been experimenting with something like that for a project we’re doing on DevOps.com where, you know, we want to have web people do a little bit more hands-on playing with different vendors’ tools and so, we’re trying to give them a sandbox, right, where they can go in and do these things.

    Because, you know, we were talking off mic, right? Originally, this was gonna be an announcement around KubeCon which, unfortunately, you know, got pushed. And, you know, not getting into the whole politics and the whole thing around Coronavirus and all this stuff, but certainly, we need—we need more ways, we need more diverse ways of allowing people to see what kind of new products, what new technologies, how can they get it under their nails a little bit and really—

    Spoonhower: Yeah.

    Shimel: – kind of see if it’s for them?

    Spoonhower: Yeah, absolutely. I mean, I think people learn in lots of different ways, right? Some people wanna read, some people wanna watch a video. I personally really like doing this stuff. Like, until I’ve made it, coded it, whatever, I never remember it, so.

    Shimel: Absolutely. And you’re not alone. I do think specifically for technology vendors, having those kinda hands on that people can actually go in and we see it even on our webinars, man. If we do a webinar where the vendor’s doing a real live demo that people can follow along, we get insane numbers of people signing onto that, versus someone just talking, you know, and that doesn’t work.

    So, let’s talk a little bit about what was driving this, for you guys.

    Spoonhower: Yeah. I think, you know, we—so, I should say, when I say “we,” it’s me and one of the other Co-founders. Ben and I spent a while at Google, and Google—it’s a weird place, in a lot of ways, but there are some advantages. Like, there were ways in which being at Google is a little bit like seeing the future, and they weren’t called microservices at the time, but there were a lot of teams that built software that all had to stack together.

    And we saw, you know, sort of firsthand the problems that they had in terms of like, okay, I’ve broken this thing into pieces and now how do I understand that bigger picture? We saw a lot of arguments among those teams, like, you know—“Oh, no, it’s your fault,” “No, no, it’s your fault”—and needing some data to actually kinda resolve that.

    And so, I think, for us, you know, we saw the rest of the industry starting to adopt these things through different technology, maybe, but the same idea that you can have teams work independently and deploy independently yet still build a single application. And so, the motivation was really to try to help those folks to understand what’s happening and, you know, we see LightStep being used by individuals, but actually more and more by teams. We see it used in operational review so, you know, you’re doing a post mortem, you wanna show people what happened. LightStep is a great tool to help facilitate that conversation.

    And for us, this is really about being able to help those folks deliver software more confidently and then, you know, get back to building new features and doing all the fun stuff that they’re doing.

    Shimel: Yeah, yeah. I mean, wouldn’t it be great if we just, all we did was the fun stuff all the time? [Laughter] Some of us are lucky enough to do that, me included. But this thing, these are important things that need to get done and, you know, and LightStep is following that up.

    I wanted to return to the KubeCon thing if we can, for a second, Spoons. Not KubeCon itself, but rather the Kube community, right? And you know, look, Kubernetes itself comes out of Google, right? I’m not gonna say the Google people invented microservices, but certainly a lot of very foundational work was done there, right?

    When you are out and about or you see how this community has grown and the enthusiasm for things like LightStep, right—I mean, how does that, you know, kinda give you a warm fuzzy inside to say, hey, man, this is a whole—there’s this humongous community of people who kinda share our visions or our outlooks, and it’s a lot easier to sell to, and I don’t mean that you’re a salesperson, but it’s a lot easier to sell those solutions, right, than it is—you know, it’s not like pushing rope uphill.

    Spoonhower: Yeah, yeah. I mean, I think [Laughter]—yeah, I mean, Kubernetes is a result of a lot of mistakes that were made the first time around. [Laughter]

    Shimel: Yeah. [Cross talk]

    Spoonhower: So, I think a lot of people would admit that. No, but I think the community is really great. I think the thing that’s pretty exciting to me about it is really thinking about things like infrastructure as code, about, thinking about automation, about—and, you know, these things which we kind of think of historically as, you know, maybe more of a craft actually can be an engineering discipline, and I think that’s a lot of that community. It’s actually really great whenever I’ve been to KubeCon to see the range of folks there from, you know, sort of platform folks all the way up to people that are just building features and shipping code, that they’re also excited about those tools. That’s really, for me, makes a big difference, just to see that it’s not a niche thing so much, right, that it’s really a broad community that’s excited about it.

    Shimel: Oh, yeah, no, it’s broad. Me personally, I have 20 years in the security InfoSec space. I get excited when I see all the security people there about it, because I really feel like maybe this time we’ll do better, right? [Laughter]

    Spoonhower: [Laughter]

    Shimel: And we got out in front of this and where, you know, security is really kinda—it’s not the caboose on the engine, per se, it’s like, people care, and not just the security people care, the community cares.

    Spoonhower: Yeah, yeah.

    Shimel: And that’s important. So, you know, in today’s world, right, this isn’t this year’s release, this is just the April 2nd sort of new feature. So, what—and one builds on the next, right? So, what’s next here?

    Spoonhower: Yeah, so, I think we’re continuing to think about how we can do automated analysis. So, we have a future in the product already called correlations which, essentially, you know, you pick out some effect you think is important, you know? Errors went up, speed went down, whatever it is, and LightStep will find the things that correlate with that within sort of the set of requests that you’re looking at. So, we have a bunch more features related to that coming out later this year.

    So, this is kind of how I—I don’t know. When people say AIOps, I always gotta roll my eyes a little bit, but this is kinda how we think about, you know, bringing artificial intelligence to DevOps, right, is that it’s really through sort of putting the right clues in front of you, right? It’s not about providing 10,000 clues to solve a mystery, it’s about 5, right?

    Shimel: Right.

    Spoonhower: So, a bunch of stuff around that, and then—I mean, the other place that we’re thinking about is how do we get more proactive, right? How do we get in front of you and help you anticipate when these things are gonna be happening as well?

    Shimel: Absolutely. So, I suppose one day you’re gonna grow old and you’ll be on this side when—so, I’m no longer building products or founding companies that are building products, and I hear public lands come up with all these…and I don’t wanna say it’s just some marketing term, but—

    Spoonhower:[Laughter] You can definitely say that, that’s fine.

    Shimel: You know, I will tell you, the first time I heard AIOps, too, I said, “What?” You know? But, you know, I’ve been in technology long enough where—look, just don’t call me late for dinner, right? I don’t really care any more what you call it. And also on the media side of it versus on your side of it, I just report the news, you know, theoretically, right? We don’t—if you wanna call it, you know, if it quacks like a duck and it walks like a duck, it’s a duck.

    But in any event, I do hear you about that. And, you know, you wanna talk about an overhyped term? AI?

    Spoonhower: Yeah. I think in the world of—in DevOps, I think it’s actually, there’s some—

    Shimel: It’s overhyped, itself.

    Spoonhower: Yeah, there’s some [Cross talk] though, I think, because you know, if you’re talking about, like, I don’t know, ad placement or something like that, no one really cares about why, right? Like, you put the ad and someone clicks—nice job, right? Like, good job, robots.

    But like, you get paged and something’s like, “It’s that service over there, you should definitely roll it back.” And you’re like, “Yeah, but why?” Like, I’m not just gonna start rolling back things because robots tell me to.

    Shimel: Yeah, “I said so, that’s why.”

    Spoonhower: Right. [Laughter]

    Shimel: [Laughter] But I hear you, man, it’s out there. Well, listen, I told you, the time goes really quick here, and I think we’re about out, man. But Spoons, it was great having you on here. Good luck with this new release. I would say I look forward to seeing you at the next KubeCon. Let’s hope there is a next—

    Spoonhower: That’s right. I’ll see you somewhere, and if not, online.

    Shimel: Yeah. Otherwise, we’ll have you back here on the podcast, man.

    Spoonhower: Cool.

    Shimel: Alright? Hey, that was Daniel Spoonhower, Spoons, CTO and Co-founder at LightStep. He was just our guest on DevOps Chats. This is Alan Shimel for Container Journal, DevOps.com, Security Boulevard. You’ve just listened to another DevOps Chats. Have a great day.

    — Alan Shimel

  • Bridging the Network Automation Skills Gap

    Bridging the Network Automation Skills Gap

    With the introduction of network programmability and APIs for everything from management systems to devices themselves, the modern network landscape has evolved to being managed like software. This shift from hardware-focused networks to software-centric functions has had a profound effect on network management techniques and skillsets required to keep pace with the changing ecosystem.

    A recent EMA survey notes that skills gaps associated with network automation are an issue for 96% of enterprises. Even respondents with significant network automation initiatives in place felt their solutions stretched the abilities of their network teams. Network automation can expose skills gaps where network engineers might lack experience with new tools and software development. At the same time, application development teams, who likely would be included in building an in-house network or automation software solution, will also lack networking skills. Hybrid skills with both development and networking expertise are rare and available only at a premium.

    This journey to network automation can be difficult, especially given the required technical skills that are not in abundance within many organizations. Simplifying network automation and empowering a broader set of participation through capabilities to easily create and manage automation workflows presents a key opportunity for organizations to not only mature efforts around network automation, but to also execute against broader business digital transformation objectives.

    Four Keys to Democratizing Network Automation

    The key to democratizing network automation and bridging the skills gap is to provide NetOps teams with the ability to easily design and build network automation without having to re-tool, build custom code or learn specialized software skills. There are four key elements required to effectively democratize network automation in the enterprise.

    Easy Onramp for Non-Developer

    Automation is typically accomplished through writing code in a particular programming language. For a person without any programming skills (or even minimal skills), they must first learn how to code before they can even attempt their first automation task. This is very difficult in terms of time and interest. Additionally, many network engineers see the amount of work it would take to learn to write code, and become disinterested. There has to be a new set of tools and methodology to empower these people to automate their work without the long road of learning to become programmers. It’s important to invite as many people to the automation table as possible including IT, security, cloud and NetOps teams, to streamline business processes.

    Capture Network Tribal Knowledge

    Within the organization, there is already existing embedded tribal knowledge that is incredibly valuable. By harnessing and transferring that knowledge into an automation workflow without a lot of programming, it provides an accelerated path to network automation. The goal is to leverage traditional network automation and management practices (Scripts, CLI) for re-use across the organization.

    Network engineers with years and years of experience have a valuable set of processes and procedures they’ve built up inside their heads, and this wealth of information is a valuable asset. Unfortunately, this information is rarely documented and is normally passed on by activity and word of mouth. This is the type of tribal knowledge that needs to be captured, preferably through automation workflows that can be used in a task, documented in the workflow and modified over time to improve it or change it as the infrastructure changes.

    Integrate DevOps Principles

    Augment network automation with CI/CD pipelines and API services. These are well known solid principles that are already used in other domains outside of networking and pay big dividends when applied to the automation of complex networks. When we look along the horizon of the network automation journey, these same CI/CD practices will be adopted by future networking teams and allow them the same advantages of reliability and velocity within the network.

    Enable Cloud Architect Participation 

    It is important to have networking people that can understand the networking constructs of the cloud, as well as cloud teams that understand how it integrates and operates within the network. Enabling cloud architect participation in network automation will allow for the real-time feedback and control of cloud-based applications and infrastructure critical to modern networks.

    Cloud networking is increasing in complexity and will soon require the skills and experience of network engineers to help solve the problems of cloud networking. Traditionally, cloud and networking teams have been separated because their technology domains have been siloed and there was very little interaction required between the teams. As technologies like SD-WAN have been adopted, the line of responsibility between cloud and network teams has blurred. Cloud and networking teams will need an automation platform that can be used by both to create workflows and automate between both domains. Business technology decisions are forcing these two groups together, and they must have a way to use their own tools, but also have a platform to collaborate automation between them.

    As applications and services are becoming more complex, distributed and require connectivity and policy enforcement across a diversity of domains (such as cloud, SD-WAN, data center, wireless and network applications), the management of these network concepts requires us to re-think how we have traditionally managed networks. As network automation plays an increasingly larger role in enterprise networks, a byproduct of the complexity is a skills gap that threatens to slow or stymie enterprise initiatives.

    By making the onramp to network automation easier for non-developers through capturing tribal knowledge, integrating DevOps principles and enabling cloud architect participation, enterprises better ensure a successful migration to network automation across multiple domains.

    — Rich Martin

  • DevOps for the Rest of Ops

    DevOps for the Rest of Ops

    The IT pro gig usually includes a basic headwind. They’re part of a cost center, and budget for resources and staff are limited by design unless they’re managing a strategic initiative, which—at least for a while—is more generously funded.

    Early adopters of DevOps, Agile or site reliability engineering (SRE) teams have been budget beneficiaries when tied to transformation projects—as has always been the case for high-visibility IT initiatives. Fortunately, DevOps has proved more helpful and stickier than some other “next big things,” and for some organizations it’s the engine behind growth and even actual digital transformation success.

    IT pros—pragmatic as always—have taken note of DevOps potential, even while keeping lights on, systems up and reliably delivering the critical services businesses have depended on for decades. The question for IT leaders is, how can IT teams realize some of the benefits of DevOps, while operating inside traditional IT organizations?

    As DevOps has finally slid off the hype peak and into the shed of useful tools, experience has distilled out a couple of conclusions. First, the tenets of DevOps—breaking down walls, shared feedback loops and learning to love automation—don’t only apply to new or cool apps. They’re even more helpful in traditional IT. For example, the rise of automation isn’t driven by a conspiracy to replace admins with robots, it’s a necessity to shrink the toil of manually wrangling all the new complexity of cloud and cloud-native tech on-premises.

    Second, many of the ideas behind DevOps and resulting practices are a direct attempt to fix issues in IT that have frustrated IT teams for years. Ask any IT pro if they’d like to move all deployment to Monday through Thursday, 9 a.m. to 4 p.m., instead of midnight Saturday nights. That team will go nuts for CI/CD. Better, it turns out you don’t have to do all the DevOps things to get many of its advantages.

    But even if your business still operates from the C-suite down, entraining quarterly budget spending, waterfall models and lots of walled silos, there’s an opportunity to adapt DevOps processes and culture to deliver progress on truly transformative initiatives to help grow the business. The trick is figuring out which DevOps practices are valuable and which ones are more useful to developers.

    Luckily, focusing on just four can get your team most of the benefits without (much) disruption, and development teams are eager to share its growing body of new techniques and successes. We’re working in an interesting epoch of IT history, a potential democratization of DevOps for everyone, DevOps for the rest of ops.

    Continuous Delivery

    Until recently, some IT pros were skeptical, even suspicious that the development teams’ goal has been to automate them out of a career. Others felt developers were demanding everyone to learn how to program full-stack and invent a way to move fast and break things just as they can (more safely) do. A few admins have even had unpleasant encounters with some DevOps proponents when they showed interest. Nobody likes reflexive replies, such as “You use Oracle and DB2? Switch to MongoDB and come back later” to eager 101 questions, even if they’re just from a handful of community members. It’s especially unpleasant for developers and IT pros alike because that’s not how operations works. We share, we’re open, we’re helpful. We came to tech to help end users get more out of technology.

    Instead, successful teams often start by identifying tools supporting fundamental operations goals, without the distraction of any particular technology or vendor, and seek buy-in from both developers and ops. Because if you’re in operations, you’re the team that must:

    1. Find a way to get software into reliable production.
    2. Maintain functional, useful governance.
    3. Maintain apps for years more than anyone thinks they’ll possibly be online.

    Operations knows this because it has always been the last mile of delivery and deployment, it cares about availability and SLAs, and defines process for breakfast.

    One of the first tools associated with DevOps is CI/CD pipelines, which are literally implementations of automated policy about how code goes from origination to production. But, overlooked in many DevOps blogs is the last step of CD because it’s not for the faint of heart. That’s unfortunate—operations is good at figuring out that last mile.

    When operations teams embrace CI/CD/Continuous Deployment, they’re rewarded for investment in automation in the best possible way: less night and weekend work. As a result, the pipelines they build aren’t ephemeral experiments but production-grade CI/CD because it’s part of production itself. And as change velocity increases and error rates decrease, many teams refuse to go back to manual deployment unless they’ve exhausted all other options. Pipelines don’t replace IT pros, they improve work-life balance.

    When the operations teams present developers with standardized, battle-tested nozzles through which changes get more quickly deployed with fewer errors, the reaction is usually excitement. It allows developers to mate their build, unit test, packaging and delivery head ends to the typically complex delivery/deployment operations pipeline. Best of all, ops teams tend to include monitoring, security and even cost governance into their processes, resulting in effective controls being automatically and reliably applied. Further, ops metrics create and enrich the feedback loops dev need to improve code over time based on real-world performance, not dev or test data. In environments like that, culture change is more organic based on shared goals, success and the flexibility to let each team do what they enjoy most.

    Start Breaking Stuff Fast

    Operations teams can have a little envy when it comes to breaking things. They’re technologists just like developers, and like most people attracted to technology, they like to learn by disassembling, recombining and otherwise hacking tech to learn the intricate details of how it works. But unlike a developer testing code, experimental configurations or other normal laptop environment one-offs, operations absolutely can’t experiment. IT is risk-averse by commandment, for good reasons.

    “Get a lab!” sounds like an obvious suggestion, and it is—but it’s a great example of easier said than done. Because IT is a cost center, labs are often seen as duplication and waste because their value is harder to justify in a budget presentation. But with a little unconventional use of the monitoring data you already have, that’s not necessarily true. Operations regularly correlates change events with performance histograms in the ongoing search for improved reliability. Consider adding before and after metrics to changes tested in a lab environment first. You’ll likely find you can prove the more realistic the pre-deployment testing, the lower the error rate and higher the performance of that system.

    But the mechanism behind improved quality isn’t the lab itself or data it produces. It’s the team’s experimentation—and some blood, sweat and tears shed—impossible in a production environment. IT also allows operations to invite developers to go on deployment and tuning ride-alongs. Developers can use whatever tools they prefer, make changes, probe and otherwise disassemble the lab stack as needed. And from a communications and culture perspective, this can be helpful.

    Operations lab environments also allow IT pros to separate hype from reality on their own. They can install the latest shiny new tools for evaluation, try new methodology or even test significant platform architecture redesigns. In short, they can break whatever they want without affecting production and provide both feedback to developers and a place for dev to experiment with changes. Here again operations can be the lead, with new avenues for communication, cooperation and culture change.

    Monitor Your Success

    Most developers aren’t data driven, they’re functionally driven. They love data-informing design and are good with tools that wrangle it into something useful, but at the end of the day their job is to build code that does things. IT pros have monitoring in the DNA because detailed performance data collected over time is how they initially validate hunches but also break down conclusion bias. But while everyone seems to understand the value of metrics and data, it’s not usually considered a tool for DevOps communication.

    As mentioned earlier, feedback loops are a fundamental principle of DevOps. Changes go into production, production generates data sent back to developers to guide future changes to further improve service quality. It’s a virtuous cycle, and both teams value it highly. The trick is in creating a language between dev and ops about what to collect, how it’s used and the stewardship of feedback over time. Step one is to sweep away preconceptions on both sides and start fresh.

    A common approach is to bring a small—one-pizza small—team together and start with a blank whiteboard. The goal is to matrix the inventory of what’s already available with what’s helpful, then identify gaps for both teams. Development teams are hungry for the incredible richness of coverage operations can offer, while operations is eager to mine new instrumentation the development team relies on for troubleshooting and internal application performance monitoring (APM).

    APM for hybrid applications is a great example of both teams learning how to create ensemble visibility into hybrid on-prem/cloud-native services. High performing operations teams ensure they add metrics dev teams need, and developers discover that the “impossible” information to debug pesky apps has been collected for years and is an API call away. Once again, a little conversation can lead to relief, and earned changes to hearts and minds.

    Reward Success With Big Dreams

    The final common experience of developers and IT pros both are never really done. Code can always be improved, and service delivery can always get better. But because we’re human, sometimes we get tired, stuck or bored, and it’s easy to take a break, especially after a big win. That’s natural. There are years of deferred work and technical debt lying around, and no time for perfectionism, especially in enterprise IT. This critical aspect of DevOps needs to remain a focus of any team adopting DevOps techniques—remembering DevOps is never done. It’s a culture of continuous, collaborative improvement and evolution. This shared understanding sustains the bonds between dev and ops.

    When teams innovate, when they continue forging closer relationships and when they share improvements as the team culture changes, they start asking bigger questions. We’re conditioned in IT to accept that certain things can’t be achieved because, reasons. As a result, sometimes suggestions and questions that could finally resolve the team’s most bedeviling challenges might not always be made or asked.

    Track your “impossible” ideas which drive progress, then promote successful ones while sharing details of why they worked. This in turn helps socialize the value of success resulting from openness and encourages great ideas that may have been sitting on the bench for a long time.

    Again, you’re investing in more metrics and signals, but not limited to performance data. When you dream big, especially as you untangle the enormous expectations of digital transformation into operations changes, you’ll start to rely on new metrics. For example, business metrics tied to corporate objectives, behavior signals accurately reflecting CSAT, change and error rate data helping teams decide when to add new versus when to fix and new ways to assure leadership operations have everything under control. When managers are confident IT pros can safely transform gear, services and staff without interrupting the business, the most sought-after DevOps goal can come into focus: A new freedom to innovate at speed.

    — Patrick Hubbard

  • DevOps Institute Upskilling the Enterprise

    DevOps Institute Upskilling the Enterprise

    DevOps Institute released its annual Upskilling the Enterprise survey and the results are as interesting as last year. The organization of DevOps teams and its interaction with speed/success was one area that intrigued me, and sent me off looking for more information on the topic. This led me to this interesting site that does a great job of delving deeper than “We’ve all heard of Conway’s law.”  Good descriptions there on what works and why that will be useful to many organizations. The only true weakness in the content that I found was the “vacuum syndrome” that doesn’t delve deeply enough into what’s right for your organization, which certainly impacts your choices.

    DevOps Institute’s discussion of soft skills and DevOps roles being hybrid is also interesting. It creates a bit of a conundrum. We all wish the technical genius was also a charming personality, but often that is not the case. In the end, our job is technical, and we need to deliver technical solutions. There is a balance between making the genius more personable and finding personable people capable of achieving the goals. Considering that hiring is currently a struggle, we often do not get to choose. Training can help a lot, and in the end, finding a balance that works for a given organization is going to be a bit of work for management.

    What comes as no surprise is the finding that CI/CD are the entry level required skills for DevOps. No matter how far along on your DevOps journey you are, CI and CD are core to the job at this point. Expect that to change over time as deployment and security become more core and commonplace. In the end, great CI/CD is not much help if deployment and monitoring are weak, so the tools that win these battles will rise in importance over time without taking away from CI/CD.

    Their idea of the E-shaped human is interesting, but in diagramming, some detail is lost in the abstraction.

    I once worked with a super-genius DBA that could do anything with the database. We (managers) met with him on issues and let him tell us how impossible it was and that we were destroying the enterprise’s data integrity. Then gave him a day and let him tell us how it would get done effectively without major disruptions. Only after we’d met with him the second time did we move forward with the project, talking to business owners. Essentially, we became the wall between him and anyone who would be put-off by his face going red and his angered declarations. It was an acceptable arrangement, and he never let us down. I worked with him on dozens of projects, and only once did I see him exposed to outside consultants (against the wishes of several of us), it was ugly, and they pegged him as a stone-waller. He wasn’t, just needed to vent first.

    This person was responsible for the stability of our data, for upping the volume of throughput when needed and for making sure SQL didn’t get past DBAs without being safe. Would we have liked him but with a sunny disposition? Sure, but given a choice, I for one was happy with him and an angry disposition. I worked other places I wished I had someone with his unparalleled skills on the DBA side. So, all of IT needs to get better at interpersonal communications in a DevOps world, but “better” is relative. I’d still keep him in the back, making sure our systems got what they needed.

    The survey head-nods to this reality where soft-skills came in third as important skills. Still high on the list because it is important, but it shows that technical skills are why we have the job. Soft skills are how we do the job with less friction.

    Considering that I am the communicator/negotiator, it might sound strange for me to stand for tech skills first, but that is who we are–technicians. Highly skilled and highly compensated, but technicians. There are few (as a percentage) people who can handle both technical and communications, but by the same token, there are few people who can be an absolute boss at their technical responsibilities and see each unexpected event as a challenge to be overcome. I love to learn new things, and at that job I learned a ton of Oracle stuff from our tech genius. But I would be foolish to believe I’d ever match him in both domain and tech knowledge when it came to databases and our employers’ needs.

    Anyway, I picked three of the points that interested me to expound upon. The entirety of the report is definitely worth a read to get a feel for where others are on their DevOps journey and offer some ideas for yours.

    Meanwhile, keep kicking it. We’re in the midst of COVID-19, and I hear you, some of you are simply buried in organizations that weren’t prepared, others are suddenly looking for work because your previous employer is suddenly not a viable business. Here’s hoping it gets better for each of you, while the rest of us struggle on.

    — Don Macvittie

  • Four Metrics Every Engineering Manager Needs to Monitor for Success

    Four Metrics Every Engineering Manager Needs to Monitor for Success

    Engineering managers and leaders are charged with a challenging task every day: How do they support engineers to do their best work, and how do they know if their engineers are doing their best work? 

    The former is easier than the latter. To enable an engineer’s best work, you’ll want to make sure they are clear on their goals, focused on the right priorities and have the information they need to do their work. A critical role of the manager is to eliminate any blocks that get in the way, too.

    Yet, despite those efforts, engineering managers struggle to identify if their engineers are actually doing their best work. Should they measure how many items they complete? Or how much code they write? Or the quality of their work?

    All of these are relevant puzzle pieces, yet they’re hard to objectively measure, and don’t indicate the full picture of an engineer’s work.

    One revelation from our productivity research at Uplevel is this: Instead of focusing on output, managers can find more success in focusing upstream. When we consider the conditions that support or block work in the first place, we give engineering teams the best chance to do their best work.

    Here are four upstream metrics worth tracking.

    Time to Focus

    While it seems pretty straightforward to focus, we live in a distracted world of work. Office layouts are open, calendars get filled, Slack sends notifications all day long (and thanks to APIs and integrations, you can be notified about anything from the status of a Jira ticket to when lunch will be delivered). Effective engineers need uninterrupted blocks of time—at least two hours or more—to do deep work and focus on challenging problems. A manager can help enable this in two ways: by serving as the shield so that the team is in fewer meetings, and by creating a culture where developers feel empowered to block off focus time, either individually or as a team.

    Meeting Volume and Distribution

    Managers can ensure that their teams meet when they need to, but no more than that. These meetings are needed so that people can collaborate, problem-solve and follow engineering-related rituals in the world of agile development. However, it is easily possible to have too many meetings, or ineffectively run meetings. When occurring at inconvenient times, just two meetings a day can take up most—if not all—of a developer’s deep work time. Monitor your team’s meeting levels and distribution so that there’s optimal time for deep work before, after or in between.

    Avoid Multitasking

    If your developers are going to be in a meeting, it’s important that they are focused on the goals of that meeting—and not multitasking. Managers can lead by example and keep laptops closed, unless they need to share or present. Past and emerging research points to evidence that manager behavior shapes individual contributor behavior. That role modeling can lead to new habits, good or bad.

    Too Much Context Switching

    Are your developers focused on a small number of high priorities? Or are they jumping across epics, project phases, focus areas and code bases? Keeping each developer focused on just a few areas helps reduce context switching—a costly obstacle to effectiveness. Every context switch—like, say, shifting focus from a 10-minute standup, to a cluttered inbox, to a fire-drill Jira ticket—requires time to switch out of the first context, into the new context and back to what they were working on in the first place.

    — David Youssefnia

  • IT Services and DevOps: Friends, Not Foes

    IT Services and DevOps: Friends, Not Foes

    Development and operations, aka DevOps, is the marriage between software development and IT operations. The goal of a DevOps team is two-fold: to more quickly make products that are better for customers, and to align more closely with business goals.

    While it sounds pretty great, something doesn’t sit well. What about your service department?

    Service department representatives often become wary at the mention of DevOps. Why? They don’t want to be left behind or lose their seat at the table. Both DevOps and service teams work toward better customer experience, so how do you keep your services relevant with DevOps at the wheel? Will DevOps be the end of IT service management (ITSM)?

    To answer that last question: no, DevOps isn’t going to end your service department. The fundamental goal of having a DevOps team is to improve the alignment between how IT products are created by development and how the operations staff run them once they’re in the real world.

    It’s excellent that DevOps wants to be more efficient, but your services have to be able to keep up. How will you match services to customers’ demands if the DevOps team goes its way? The solution is easier than it sounds, and there are real benefits to the ITSM-DevOps collaboration.

    Creating a Complete Story

    What’s your service department’s main goal—helping your customers? And the primary purpose of DevOps? Are you helping those same customers? One reason a DevOps team exists is to improve customer experience through better alignment across the organization. But, customer experience is your service team’s area of expertise. So, what do you do?

    You hop on the DevOps train and change the destination to the ITSM-DevOps department. DevOps is about breaking down silos. The word says it all. You go from the development team and the operations team to a single team working toward the same goal. The DevOps team is all about the product, and your team is all about the service. But, what if the DevOps team makes products you can’t support with excellent service?

    Once again, the solution is to work together. As you’re breaking down silos, break down another wall—the wall between product and service.

    Bringing ITSM and DevOps Together

    Although ITSM and DevOps serve the same customers, they work very differently. DevOps creates and manages the product with the goal of keeping things running. They are responsible for the product as a whole but usually focus on a few changes or improvements at once. They often work according to Agile methods.

    In ITSM, however, you’re always getting questions about everything. Your work is unpredictable, and while Agile has a lot to offer, you also need guidelines to create more structure in your team’s work. You probably use the Information Technology Infrastructure Library (ITIL) framework, which may seem slow and complicated to your DevOps colleagues.

    How do you go about uniting the two teams?

    Service desks face two main problems when they aren’t that close to DevOps. The combination is paradoxical because DevOps tends to move too fast and too slowly, depending on who is asked and when something is asked.

    Here’s what happens: The DevOps team has its own pace and plans, working on products and features according to the demands of key stakeholders. The service desk is the place where your customers go to get their issues fixed or to gather help with existing features, and when existing features don’t provide as needed.

    The two tracks aren’t always in sync. For instance, DevOps releases new features faster than they did before development and operations teamed up. That means your service agents are receiving questions they can’t answer yet. Plus, the answers are always changing, so they can’t rely on experience.

    Simultaneously, it can feel like you’re waiting forever for DevOps. You may get calls—even recurring calls—about features that need fixing or functionalities that you don’t offer yet. And receiving help from DevOps can sometimes take time because they are currently working on another part of the product.

    Getting in Sync

    Bringing together DevOps and ITSM can seem like a pacing issue. But you don’t need to work on the same things at the same time, and the question isn’t which branch should be in control.

    Collaboration requires communication. DevOps is excellent news for your service desk, provided your service team knows what’s going on. DevOps should always keep ITSM updated on what they’re working on and expected release dates.

    At the same time, service teams should always communicate their needs to DevOps. If customers have issues you can’t solve with the means you have available, make sure DevOps knows and provide them with the information they need to understand what customers are asking for, prioritize projects and provide a better product.

    DevOps shouldn’t be feared. There is no battle for survival between DevOps and ITSM. Everybody will win when they work together.

    — Jeroen Boks

  • We All Have to Have Standards

    We All Have to Have Standards

    I was working on an e-book recently about the state of the market for certain DevOps toolsets, and got really interested in how standardized organizations are becoming. We have been on this roller coaster of changing technology and different teams making different choices in language, libraries, platforms and toolchains. We all knew that couldn’t go on forever, but I started to really get into the idea of “How much standardization is being introduced?” While talking with vendors about their experience, it seems that enterprises are finally attempting to standardize, at least in the platform and toolchain areas.

    Not the type of “technology by edict” standardization we used to see, but a checklist of “If your project fits these criteria, use this platform. If it fits those criteria, use this toolset for your build chain.” Which is a good compromise between the days of “We’re a Cisco shop” or “We’re an Oracle shop,” and the days of “Yeah, first developer on a project chooses the language, database, etc.” You limit the amount of technical debt and vendor/OSS project dependency, while offering teams options for how best to get the job done.

    I’m excited about the change, and it is one of the things driving much-needed consolidation in core DevOps markets. I won’t go beyond that, as you’re supposed to read the e-book when it gets released, but it’s an exciting time to be in DevOps. We’re getting more and more tools, better integrations and even some full lifecycle toolsets starting to become clear.

    Our shop is small, and we’ve exercised standards all along, just because we don’t have the time for everyone to train on five different CI products, so we use one. But for larger organizations with more diverse needs, standardization will offer savings in both maintenance and cross-training.

    Consider that at one point there was no standardization on vehicles–everything from tires to engine oil was custom to the brand you owned. Now, it is quite often the case that the engine in your vehicle is the same engine as in a competitors’ vehicle (VW using Saab engines for years is a good example), and that means maintenance is streamlined. Tooling with onboard computers makes maintenance even faster, much like tooling monitoring/reporting is doing for the DevOps iteration. It’s a good time to be in the field, watching as things unfold.

    Stay aware of what your vendors are doing, and what their competitors are doing. One of the reasons DevOps exploded is because tools and processes became static and weren’t as productive as they could have been. Don’t repeat that mistake, keep an eye on rapidly changing developments and adopt what suits your organization. You’re rocking it now, take a bit of time to make sure it stays that way.

    And consider standardizing some of your toolset. The fewer tools to maintain, the more time to work on productive issues. And the fewer toolchains a DevOps team member has to know to be productive on multiple projects, the more productive they can be. Agility applies here also.

    — Don Macvittie

  • How to Build Creativity into Distributed DevOps Teams

    How to Build Creativity into Distributed DevOps Teams

    Today, more people than ever work remotely or are part of distributed teams. The DevOps world is no different. While working remotely has definite advantages, it also poses distinct challenges for teams trying to build strong internal cultures, coordinate product launches and hit shipping deadlines. It’s not enough to just slot in a communications app anymore. Teams need tools and processes that foment creativity and the frictionless sharing of ideas.

    Leaders know that the greatest ideas often come at the most unpredictable moments. With distributed teams, the risk of losing those great ideas in the shuffle dramatically increases. Given these realities, how can DevOps teams replicate the spontaneity and creative energy of an in-person brainstorming session when employees are located in different places around the world? 

    Get Personal

    It’s human nature to withdraw when feeling uncomfortable in a new group of people. Be it a party or work function, people will choose not to participate in conversation if it risks embarrassment among strangers. The same thing can happen due to the seemingly cliquish nature of certain distributed teams. If a new employee doesn’t know anyone personally, it is imperative to bring them into the fold and establish comfort as soon as possible. Then they’ll feel more confident sharing ideas.

    For extroverted people, this rarely poses a problem. However, introverted team members may not take so easily to unstructured casual conversation, and will need to be drawn in. This is where a bit of planning can go a long way.

    Leadership should start every team meeting with an icebreaker question that draws people out of their shells and reveals new aspects of their personalities. There is no need to get too intimate. These questions can be playful and funny. For example, asking people about their favorite hobby, restaurant or vacation destination is an easy way to kick start a conversation. Questions about long-term personal or professional goals—perhaps completing a marathon or learning to code in new languages—can also spark meaningful conversations or potential mentorships. The goal is not necessarily to force friendships (though that is not discouraged) but to make everyone comfortable interacting while knowing the company is a safe space for the expression of new thoughts, ideas and strategies. 

    Use Both Live and Asynchronous Video

    Modern DevOps relies on a high degree of automated processes. With modern pipelines that deploy code live several times a day, it becomes incredibly important to have a tight communication loop across the DevOps, Site Reliability Engineers (SREs) and development teams. In the unfortunate event that there are production issues, it becomes crucial to engage in meaningful conversations with team members and stakeholders while working on a terminal or actively fixing issues.

    Instant audio-visual collaboration plays a key role in creatively solving these types of problems. Text tools are clunky and slow. Plus, anyone late to the Slack channel then has to wade through piles of messages that further delay action. The ability to leverage synchronous and asynchronous workflows becomes indispensable when teams need the agility to communicate effectively. 

    With live video, details and nuances that are difficult to relay via text can be instantly shared without the worry of misinterpretation. Plus, talking is faster than typing, and allows for an easier back-and-forth when problems need to be solved quickly. Voice and video encourage free-flowing conversation. When a DevOps team spends less time thinking about how to communicate, they can better concentrate on how to solve a problem. 

    Because distributed DevOps teams often operate across different time zones, live video may not be practical if someone is asleep or enjoying life outside of normal working hours. This is where recorded video has special advantages. For example, recorded videos express context clearly and more effectively than sending an email when dealing with “show and tell” walk-throughs, bug reproductions, release audits, commentary on visual assets or user feedback. A lightweight video recorder can also aid in memorializing and documenting fast-changing deployment models, pipelines, scripts and configurations. These videos excel at capturing a thought in an instant, bringing energy and clarity to a conversation.

    Make Flexibility Fundamental

    With a DevOps team scattered across the world, standups and regular team meetings often take place during odd hours. Someone might be joining during morning coffee while another hops on after putting their kid to bed. Now think about how creative you feel in the morning versus the evening. If a scheduled check-in constantly fell during a time you felt mentally and physically drained, it would be hard to bring your A-game. Have you ever successfully brainstormed first thing in the morning, when hungry or right before bed? 

    Regularly switching the times of these check-ins is key. This way, leadership can show DevOps that it understands some hours of the day are just not as conducive to creativity and brainstorming as others. By accommodating employees through switching the call times weekly or monthly, leadership gives each team member the opportunity to shine. Plus, this approach gives people a fresh perspective to share unique ideas.

    Random, on-the-fly check-ins can also be a boon to remote developers. Hopping on the phone or a video call when hitting a snag can help quicken the solution process. After all, two (or more) heads are better than one. This can also relieve some of the pressure associated with sharing a new idea in front of the whole team during a formal meeting. Perhaps just bouncing a nebulous thought off of a trusted coworker is enough to bring that idea from half-baked to reality. This is certainly faster than a Slack message or email chain.

    The ability to encourage creativity with remote teams is only growing in importance as more and more people work offsite. Building a strong, open culture where team members know they can express themselves freely without fear of negative repercussions is paramount to successfully navigating these changing workplace environments. Hence, we strongly urge that modern DevOps concepts, new tools, strong releases and tight team coordination must jointly evolve to help establish a strong culture and bolster the importance of nurturing creativity in-house.

    — Badri Rajasekar