devopsy.kinsta.cloud

Category: DevOps Practice

  • Managing Data Risk in 2021

    Managing Data Risk in 2021

    You can’t protect your data if you don’t know where it is stored. The first thing to consider when creating a risk-based approach to data protection is the ability to identify and prioritize data, and control who has access to it. But, how do we do this? How do we prioritize data?

    In this TechStrong TV episode, Ron Bennatan, senior vice president and general manager of data security at Imperva, joins Mitch Ashley to discuss privacy, compliance and managing data risk in data lake environments in 2021.

    The video is immediately below, followed by the transcript of the conversation. Enjoy!

    Transcript

    Mitch Ashley: I’m very happy to be joined by Ron Bennatan, who is GM of Imperva Data Security. We’ve talked before—Ron, it’s good to be speaking with you again.

    Ron Bennatan: Hey, Mitch. Nice to see you again.

    Ashley: As always, yes. Data and security—two of my favorite topics. [Laughter] Welcome to 2021.

    Bennatan: Yeah, 2020 finally is over. [Laughter]

    Ashley: Ooh! You know, it takes a pretty big rearview mirror to look at that, but I’m not sure I wanna look back too much. [Laughter] Well, we’re gonna be kinda talking about looking forward into this new year. First, would you introduce yourself, tell us a little bit about you, and also a little bit about Imperva?

    Bennatan: Yeah. So, Imperva is kind of the market lead for data and application security. We secure all paths to data. So, it kind of straddles between both covering how data is accessed and where data is accessed, kind of covering everything regardless of on prem and cloud, hybrid cloud.

    Ashley: Mm-hmm.

    Bennatan: And I am the GM for the data security side, and really glad to be here.

    Ashley: No small problem you’re solving. [Laughter] Data is everywhere, you know?

    Bennatan: One problem, yeah.

    Ashley: [Cross talk] data. At least not very many of ‘em. So, let’s talk about it. I mean, you know, we have the whole government breach in 2020 Russia—you know, Russia, Russia, Russia, that whole thing.

    Bennatan: Yeah.

    Ashley: It certainly brought a lot of attention to supply chain attacks. Of course, you know, that being an intelligence brief, it’s collecting a lot of data, right? You know, that could, of course, happen on a—it is, did happen on corporate networks, it could also be for financial gain; it doesn’t mean it has to be a national state attack. So, I imagine that’s gonna be a—supply chain is a word we’re gonna hear a lot about this year.

    Bennatan: Yeah, I think so. I think, you know, as a, just an individual, my fear, by the way, is—you know, there’s a really big project coming for the entire world, which is the delivery of the vaccines, and that itself is a supply chain. I just hope nothing messes up there, you know? Like, you know, knowing that whatever vaccine you’re getting is actually the valid one and not—you know, there’s a lot of…everything is in data and everything at the end sits in some kind of a, like, what’s the lineage of something? What’s the providence of something?

    Ashley: Mm-hmm.

    Bennatan: You know, it doesn’t necessarily—you know, we in the data governance area talk a lot about providence and proving the correctness of data, but you know, it goes all the way to the vaccine as well. So, I think this issue that we’re seeing, whether it’s breach—breaches get headlines, okay? And, you know, so this SolarWinds thing gets a big headline, the thing that these attacks on MySQL with ransomware, that gets headlines. But really, the bigger issue is getting control of our data and creating kind of a complete risk oriented approach, not just plugging a hole here and plugging a hole here and plugging a hole here. It’s—that’ll never work, right? It’s much easier to create a new hole than to plug all the holes.

    Ashley: The whack a mole strategy. “Here’s another mole to whack.” [Laughter]

    Bennatan: The whack a mole—yeah, the whack a mole doesn’t work. Whack a mole doesn’t work. And especially today when the complexity of the data environment is so much greater.

    Ashley: Mm-hmm.

    Bennatan: You know, 20 years ago—okay, 40 years ago, we just had some dashboard on the mainframe. Twenty years ago, we just had Oracle sitting on some Unix server. Today, it’s such a complex environment, data moves from place to place, you know, you look at what a data lake implies. It’s no longer a single stack, it’s a combination of stacks. Whack a mole doesn’t work.

    Ashley: Mm-hmm.

    Bennatan: So—and most companies understand that. Most companies are in this process of investing in a non-whack a mole approach, in a risk based approach that says, “Okay, I need to know where my data is. I need to prioritize which. You know, not all data is created equal. I need to scan, I need to understand, I need to know who has the ability to access something. I then need to know who is accessing something. I need to control what they’re accessing.”

    So, you know, I think there’s a better understanding that whack a mole doesn’t work.

    Ashley: Mm-hmm.

    Bennatan: I think 2021 is still gonna be a transition year. I don’t think we’re gonna necessarily get there, but we’re gonna transition.

    Ashley: We won’t have universal enlightenment [Laughter] on day—

    Bennatan: It just takes time. I mean, it takes time.

    Ashley: It does.

    Bennatan: There’s a lot of data.

    Ashley: I wanna ask you about—you did a nice job of kind of outlining, you know, identifying what data you have, prioritizing it, access to it. One of the things that I’ve always struggled with is that second step. How do you prioritize? What are the things to consider? You know, there’s obviously sorta crown jewels of my business—this is the customer data or secret sauce or that kind of things, but is there a good sort of stratification of how to prioritize data?

    Bennatan: Okay, there’s a technical answer and there’s a human answer.

    Ashley: Okay.

    Bennatan: Okay, technically, there are very good ways to do this. It takes some effort, okay? You can—you really do this based on a scoring of a bunch of dimensions, okay? So, one dimension could be the sensitivity level of the data, okay? Another dimension could be, what is the impact if something happens, okay? Which—but the reality is that we’re all people, and people are driven by incentive, and incentive is not necessarily, doesn’t always necessarily create the right prioritization. It creates whatever prioritization the incentive is structured around.

    So, I’ll give you an example. A lot of projects going on around privacy—a lot of projects. Because it’s a very, it’s a relatively new thing, less understood, and people don’t always equate the fact that we have been doing the work, almost the identical work, we just didn’t call it privacy, right? We called it classification.

    Ashley: Mm-hmm.

    Bennatan: But when you look at what privacy implies, it’s the same, but very new terminology—different semantics, different terminology. So many people don’t start from an investment they’ve already done in the last five years, they start from scratch, and the incentives are created because there’s a new role in the company. There may be a new—you know, a data privacy officer, okay, and that person has certain mandates and they drive things top down.

    And so, I think that the reality is, prioritization is driven by the projects and the funding that these projects have, and they’re also driven by breaches, okay? Because breaches are very dramatic.

    Ashley: [Cross talk] response, right? [Laughter]

    Bennatan: Dramatic, immediately created. So, I think that the reality is gonna end up somewhere kind of in the middle where things are gonna be driven bottom up and top down and they’re gonna meet somewhere.

    Ashley: Mm-hmm.

    Bennatan: But it’s not terribly complex to create a good prioritization strategy, but it’s not trivial, either.

    Ashley: Yeah, you definitely have to put some thought into it, and it’s not a textbook exercise, right? It’s all contextual.

    Bennatan: Yeah.

    Ashley: What’s important to the business.

    Bennatan: And I think you’ll find in—I mean, my prediction in 2021 is that there’s gonna be a lot of, you know, privacy is not gonna just drive a lot of these from a compliance perspective. Okay, until now, privacy is really—you know, when a company embarks on these privacy and classification projects, or at least what I see from our customers, is that it’s driven a lot by compliance, okay? They need to comply with a CCPA, they need to comply with GDPI.

    Ashley: Mm-hmm.

    Bennatan: I think we might end up having also, like, an attack flavor, okay, around privacy, which is not intuitive, not necessarily intuitive. And it has to do with the fact that, you know, part of the mandate is, you know—so, say that I am, you know, year one of my consumers and you store, I store some of your data. That means I—you have the right to demand some things from me. Like, you have the right to demand that I delete you from my system.

    Ashley: Mm-hmm.

    Bennatan: You have the right or portability, you have the right to ask me what data do I own or do I store about you? And you can come to me with these requests, okay? Different terms, different subject rights request or data access requests. You know, you can easily think about what is the effort that I, as a big company, have when you ask me to do this? It’s not a small effort. So, okay, if you—now, the nice thing is, most people don’t do that, okay? I’ve never called somebody up and told them, “Hey, delete me, okay?”

    Ashley: [Laughter] No, I think that’s probably very much an exception—maybe a rare exception.

    Bennatan: Yeah, but what happens is some people realize that this is the way to cripple a company.

    Ashley: Mm-hmm.

    Bennatan: What happens if they organize and they say, “Hey”—

    Ashley: Or protest, right?

    Bennatan: – protest, you know?

    Ashley: Moving of Facebook onto whatever.

    Bennatan: “We don’t like you, let’s organize 10,000 of our friends and we’ll all hit them at the same time.” What do you think that’s gonna do from a work perspective? It’s pretty serious.

    Ashley: Yeah, I’m guessing it’s not just press a button—that’s gonna consume some resource at the company, people resource, things like that.

    Bennatan: Yeah.

    Ashley: You know, one thing I was wondering, too, is, with this year, there’s discussion now about do we need a national cybersecurity strategy, things like that, just to mention politics, the Biden administration is talking about working with other countries to establish sort of norms of what we do when it comes to cyber whatever with each other and what is considered an attack.

    It seems like that is—maybe not this year, but that’s certainly potential for regulation of those kind of discussions, whether it comes out of any national strategy or not or could be another country that says, “Okay, now we’re gonna take this to the next level.” Like, what is our position around data? Not just privacy, but if it’s compromised, then what happens to you as a firm? I mean, it could get kind of interesting.

    Bennatan: Yeah, it could. You know, I mean—again, we’re delving into, like, opinions and politics.

    Ashley: [Laughter] Yeah, I know.

    Bennatan: I’m not a fan of—I’m not sure that something like this could go that route. But I do think that it goes other routes. So, for example, like, I’m reading a book on kind of the history of the Mossad, okay, the Israeli intelligence service.

    Ashley: Mm-hmm, mm-hmm.

    Bennatan: And you can see—and whenever you read any of these books, you see that all of the intelligence community, they have back channels, right? They agree on things.

    Ashley: Right.

    Bennatan: So, to me, if we wanna have something like that, it should be, like, the cyber czar of the U.S. and some other—you know, the big players, they have to agree. It doesn’t need to be regulated, it doesn’t need to be at the politician level, it has to kind of stay at the—

    Ashley: Kinda behind the scenes with the cyber czar.

    Bennatan: Behind the scenes with the experts at the—but, you know, there’s definitely an issue with state nations, okay? This does need to be addressed. Now, you know, it’s not necessarily that every—you know, we always think that, you know, we’re the good guys, they’re the bad guys, you know?

    Ashley: We would never do anything like what they did to us, right? [Laughter]

    Bennatan: Well, you know, there’s examples in history, okay, of things. So, I do think this is coming. I don’t believe it would happen at a politician level, it will happen at the actual subject matter experts. And by the way, there are good examples of things that have happened like that, especially in the financial services community. I mean, for years, maybe even decades, you know, FS-ISAC, they work together to create some level of standards across investment banks or banks. These things work, I just don’t think it’s political.

    Ashley: Interesting. Well, I wanted to kinda tap your thoughts on that. I know I threw the line pretty far out there in the water, [Laughter] just to kinda get your thoughts on that.

    Well, let’s reel that back in a little bit and talk—you know, there’s GDPR and CCAPE for California. A lot of people are still in this, I think in this kinda implementation stage, even though we’re supposed to have those things all covered, right? Maybe a little farther along with the GDPR, of course, but is that something, this year, you think it’s—we’re in this, okay, let’s now institutionalize some of those things in the organizations and really take care of our knitting there or do you think there’s more things on the horizon that are gonna come at data people?

    Bennatan: I think it’s gone into an implementation phase, for sure—which is good.

    Ashley: Mm-hmm.

    Bennatan: You know, there’s a lot of things, many things are not terribly clear, by the way, even though it’s in implementation. I’ll give you an example, right, this notion of being—my ability to request that you delete me, okay? But let’s take something else, which is a different type of compliance requirement, which is for you to keep logs for a certain period of time, activity logs. There are a whole bunch of other compliance requirements that require you to do that.

    Ashley: Data retention requirements, yeah.

    Bennatan: Yeah. Okay, so—now, what happens if part of the activity logs has my identity in it and I request you to delete it?

    Ashley: Mm-hmm.

    Bennatan: Should you delete it, or should you not delete it? One, in order to comply with one thing, you should delete it. In order to comply with another thing, you can’t delete it, maybe for three years. So, there’s even things like this, which are conflicting messages. And at the end of the day, it’s going to depend on some level of interpretation.

    Ashley: Mm-hmm.

    Bennatan: Which, at the implementation level, I can tell you, I can see two approaches. One is—okay, this interpretation needs to be given by legal or it’s okay for the business to give the interpretation to that. So—and you know how it is. If it goes to Legal, it takes a longer time; if the business can make the decision, then it goes much faster. And I’m seeing more and more of this moving to the business, taking ownership, and making the decisions, in which case, it goes much, much faster.

    Ashley: Well, and oftentimes, it seems that that always happens with laws and regulations, right? There’s what it is and there’s the interpretation of it, and there’s sorta those middle tier organizations, the consultants, the larger firms to sort of establish the frameworks that, “This is our interpretation of it,” so if you do that, you’re sort of in the norm with what the industry is doing. Who knows if that’s the right interpretation or not? It’s always subject to change, but that’s usually where that help comes from, at least on the big thing.

    Bennatan: Well, yeah. I mean, it starts from there and then usually what happens or what it evolves to, which makes it really go much faster, which is what’s happening now is that if—and by the way, it goes back to your question about this nations coming together or an industry coming together. The minute you have some critical mass within a certain peer group and you say, I don’t know, maybe I’m an insurance company, if my peers have taken upon themselves a certain interpretation, I usually don’t even care where they got it from.

    Ashley: Mm-hmm.

    Bennatan: Maybe they got it from these consultants, maybe they didn’t. But if three big players in my industry are doing the same thing, I’m gonna do the same thing.

    Ashley: Yeah.

    Bennatan: And that’s what’s happening now is that, you know, I know the insurance industry well and I know the banking industry well, and I can see that they’re now, “Oh, we’re gonna do like them” and then it’s much easier to follow, like, a track that seems to be proven than to try to invent it. And that’s happening and that’s why I think there’s an acceleration for a lot of these things.

    Ashley: Yeah, I agree with you. That makes a lot of sense. What are your other thoughts about 2021? What are we gonna be thinking about that we didn’t realize is gonna be on our radar for this year?

    Bennatan: I think we are going to kind of understand this notion of, you know, look at the data from every way that you can get at the data, not try to—like, historically, there have been a focus on pillars or silos, which were more and more, or in the past were driven really by tool specialization, okay? Like—okay, I’ve got the DLP and I’ve got this and I’ve got FAM and I’ve got databases, and I’ve got big data.

    And today, the lines are so blurred, especially on the cloud services, they’re even more blurred that there’s starting to be a realization that we really need to look at all paths to the data and all ways to secure the data and look at it instead of individual silos, looking at it as, “This is my data fabric.” Okay, so you’re starting to hear things like, you know, data lakes, data fabric, data services, and the data service could be structured by different implementation stacks. But you’re no longer trying to secure each stack by itself, you’re trying to secure the entire fabric.

    Ashley: Mm-hmm.

    Bennatan: And it’s really the right thing to do. It’s the right way to think about it. It is something that requires a transition, because even organizationally, many companies had one group doing this and one group doing this and one group doing this. And this notion of, you know, looking at a single overlay control layer, okay, which can drive policy down or can bring things up is, you know, it’s time for that. And I think 2021 is gonna be, part of this year is going to be starting to deliver things at that level of abstraction, and it’s also enabled by the fact that there is technology that can kind of bubble everything up into one level, right? You don’t need to look at it as 20 different tools, you need to look at it as kind of like a federated layer, and at least it’s doable, right? Ten years ago, it wasn’t doable to do any of it.

    Ashley: You know, it’s interesting. What it makes me think of is maybe some norms that we’ve had for a while of thinking about data owners, and that sort of puts the blinders on of, now I create that vertical view of, “This is that data for that purpose for that business unit for that application or process, whatever it might be.” And what you’re saying is looking at it more systemically, right? The entire collection of data, and securing it, but also uses of that data and sort additional intelligence and things you can mine from it.

    It’s interesting—so much, as we continue to move more and more into the digital economy, digital world for how we operate, all of those digital activities generate their own data, too. So, now, we’re adding that to the fabric of the data of, you know, we have a lot more usage information, we have a lot more telemetry that we never had. You know, it seems to be expanding exponentially. So, our ability to even just fathom what’s there and how it’s—what uses it might have and how it might be correlated in new ways sort of boggles my mind. [Laughter]

    Bennatan: Yeah, yeah. And, you know, I mean, it used to be this—one of the things people used to talk about when they started building big data platforms was that, you know, until recently, we just, we couldn’t use our data. We just threw 97% of our data away, and now we don’t wanna throw it away, but you’re right, it’s a beast that feeds itself.

    Ashley: Mm-hmm.

    Bennatan: But yeah, I think it’s, I think looking at the risk profile of the entire thing and not cutting into separate projects, which then have no tie to each other, right? I mean, if you want to reduce the risk, you asked about prioritization. Okay, prioritization needs to drive everything, right? Needs to drive reducing service area, needs to drive access control policies, needs to drive audit. If you start separating each one of them, not only is it not very effective, but it means you’re working 10 times harder, because you need to do this many, many, many times.

    So, this notion of a single kind of data security platform which kind of can have this internal feedback loop and use whatever effort you’re doing, it doesn’t matter if your effort is driven by compliance or security or privacy, whatever investment you make needs to build into this platform, not do itself over the next time you do anything. Otherwise, we’re just wasting our time. We’ll never get done. Because you said the data’s growing exponentially—if we’re chasing, if we’re not leveraging what we’re doing for one purpose when we wanna do the next purpose, we’re gonna lose. It’s gotta be a single platform.

    Ashley: Well, and to your point, it seems that once you sort of make that shift, right, to that platform view of data, now you can apply that to how you look at the data across all sources, all uses, all former kind of vertical siloed—you know, you may still want to look at it that way, but it’s not like, “Okay, now we’ve figured it out for compliance, but we’ve gotta go start over to do it for something else for operational process or whatever.”

    Bennatan: Exactly.

    Ashley: No, it’s there, you kinda take that—

    Bennatan: Exactly.

    Ashley: – those same tools and look at the data, analyze it. Now you’ve got that capability to do that for lots of things.

    Bennatan: Yeah, it’s exactly—it’s exactly right. I mean, you know, it’s like, in the business world, there was always this notion of a customer 360 where, you know, what’s the point that I have one system for quotes and one system for support and one system for my policy and I never know what’s going on, and I can’t upsell, and I can’t cross sell. It’s the same thing in security, okay?

    There’s this notion of data 360, data security 360 where, if it’s there, you know, then prioritization becomes easier, right? If you can check your configs and your vulnerabilities, okay, and you get this huge list—what are you gonna do? But if you know that, of this huge list, your consumer data is here and that, you know, the crown jewels are here—great, you don’t have to boil the ocean.

    Ashley: Mm-hmm.

    Bennatan: And if, at the same time, you can bring in and you can check your entitlements and you can say, “Well, you know, I have this vulnerability, and it is important data, but I did a really good job at narrowing it down and access to it is only by these individuals and access is Kerberized and—I don’t know, and maybe I also regulate the policy very tightly—these are all related, okay? So, if you have that single platform, you’re in much better shape than…you know, it’s like, I’m trying to think of a metaphor of, you know, when you have a single viewpoint, that viewpoint may be wrong. When you have—it’s kind of like, you know, these days, this discussion about whether news means anything any more, because it’s so tailored to what you wanna hear.

    Ashley: Mm-hmm.

    Bennatan: But it’s kind of like somebody forcing you to go and watch Fox and CNN and NPR and—you must watch all of these before you make a decision. It’s the same way. We can’t reduce risk by just looking at one thing at a time.

    Ashley: Yeah, we certainly have our own selection bias and data as we do our news programming as well, right? [Laughter]

    Bennatan: Yes, exactly.

    Ashley: Well, this has been a lot of fun talking with you. I just have a sense 2021 is already off to a good start with vaccines and the COVID situation. Hopefully—

    Bennatan: Hopefully.

    Ashley: – you know, we’ll round that hump and folks can really figure out new ways of collaborating with each other without worrying about COVID. Maybe that day’s down the path not too far, but we hope, anyway.

    Bennatan: Yes.

    Ashley: So, it’s been fascinating talking with you. So, we didn’t talk about, you know, go into product stuff, it wasn’t that conversation, but tell us a little bit about where folks can find out more about Imperva.

    Bennatan: So, Imperva.com, we are—you know, if we’re talking about data security, then there’s a whole section there, and you’ll see that we’re investing a lot in, you know, things that we believe 2021 and our customers are implementing. So, a lot of kind of complete security, complete security, whether it’s for compliance, security, privacy—a lot of effort in privacy, a lot of effort around data lakes, everything around cloud. Any cloud that you can think of or any data service, any data-centric workload on the cloud is something that we’ve worked very hard with the cloud vendors to provide support for. And yeah, I think 2021 is gonna be an exciting year because of this modernization that everybody’s embarking on.

    And my hope and what we’re trying to do with Imperva is that the path of modernization for the applications and for functionality also brings with it modernization of security, updated security. That’s really what we’re working on.

    Ashley: I would definitely suggest folks check you out. In thinking about our conversation before of the response to regulation and policies and things like that of looking at what other leaders do, Imperva works with some very impressive companies who are solving the data challenge from a platform standpoint. So, there’s a lot of good reasons to check out Imperva, because there’s a lot of problems you’re solving on the advance that could help a lot of organizations, so I hope people will do that.

    Bennatan: Yeah. Well, thanks, Mitch. Always great to talk.

    Ashley: You bet. Good to talk with you, Ron, and thanks, everybody, for listening in. It’s been a great conversation, and I wish all of you a great 2021 as well.

  • How MLOps Can Help Get AI Projects to Deployment

    How MLOps Can Help Get AI Projects to Deployment

    Did you know that most AI projects never get fully deployed? In fact, a recent survey by NewVantage Partners revealed that only 15% of leading enterprises have gotten any AI into production at all. Unfortunately, many models get built and trained, but never make it to business scenarios where they can provide insights and value. This gap – deemed the production gap – leaves models unable to be used, wastes resources and stops AI ROI in its tracks. But it’s not the technology that is holding things back. In most cases, the barriers to businesses and organizations becoming data-driven can be reduced to three things: people, process and culture. So, the question is, how can we overcome these challenges and start getting real value from AI? To overcome this production gap and finally get ROI from their AI, enterprises must consider formalizing an MLOps strategy.

    MLOps, or machine learning operations, refers to the culmination of people, processes, practices and underpinning technologies that automate the deployment, monitoring and management of machine learning models into production in a scalable, fully governed way. Laying an MLOps foundation allows data, development and production teams to work collaboratively and leverage automation to deploy, monitor, manage and govern machine learning services and initiatives within an organization. In short, MLOps can help enterprises see AI projects through to deployment at scale by addressing the three previously mentioned barriers: people, process and culture.

    People

    There’s a skill, motivation and incentive gap between teams developing machine learning models (data scientists), operators of those models (DevOps, software developers, IT, etc.), and the business leaders who look for the results of that work. By providing actionable visibility into the different steps of the machine learning model life cycle for these different roles, MLOps can increase collaboration across departments so that enterprises can streamline the process of getting models operational and finally meet strategic goals.

    Process

    Operations teams are geared towards ensuring overall quality of service, optimizing runtime environments on their cloud, resource management and role-based access, among many other tasks. Data science teams are often unaware of the considerations these dependencies require, and, hence, models they create do not take these tasks into consideration. Additionally, lack of a proper native governance structure as it pertains to machine learning models – with system, performance, life cycle, and user logs – stifles troubleshooting, as well as hinders legal and regulatory reporting. Organizations that don’t properly monitor their models end up introducing potential risk to their organizations. When production models don’t reflect the ever-changing patterns in data and user behavior, these factors may affect the accuracy and overall performance of the model. Machine learning-based applications are far more sensitive to production environments than typical software applications. Therefore, machine learning models need to be validated and updated over time to ensure that they’re performing properly and leveraging the most accurate and recent data. As part of implementing an MLOps strategy, by having the right monitoring processes in place to detect drift – and many other machine learning-specific metrics – enterprises can ensure their models are delivering the most accurate results, and that their projects are more likely to move forward.

    Culture

    Fear of failure (or worse) can delay organizations from collaborating to operationalize machine learning models, or prevent this entirely. Leadership needs to be clear about their commitment to the process and fully support teams in enduring the hardships involved. This is only compounded by the fact that machine learning applications typically tend to involve processes that might introduce material risk to the company. To stay on top of such transformations while still producing trustworthy value, enterprises must keep careful tabs on all model deployment processes and the updates they entail. Many operations professionals are not aware of the unique characteristics and sensitivities of machine learning. As their role is focused on managing mission-critical production environments, they are often solely concerned with the implications on their work that deploying machine learning into production might have. While these are justifiable concerns, in the final analysis, they serve as trivial reasons for significant delays in progress and deployment. The way to mitigate all these risks and get all teams aligned around these common goals is to implement a solid MLOps strategy that matches the goals and unique position of the organization and its various teams’ responsibilities.

    MLOps: All Systems Go

    MLOps allows organizations to alleviate the people, process and culture issues posed by machine learning. MLOps provides a strategy and a set of processes and best practices, a technological backbone, for managing and scaling machine learning-based services through automation. It also provides for seamless collaboration between the data science teams responsible for generating models with the teams that are accountable for running the services in production environments. This combination of process automation with role-based collaboration and governance capabilities helps pave the way toward strategic AI goals. Specifically, MLOps can help organizations move from anticipating the management and scaling of machine learning services into production and turn that into reality.

  • What Does the Future Hold for DevOps and CI/CD?

    What Does the Future Hold for DevOps and CI/CD?

    After such a challenging year, we have finally made it to 2021. Although a new year has started, we are still trying to figure out what the “new normal” will look like once the COVID-19 pandemic is finally over.

    While the pandemic has affected many businesses and our daily lives in general, it also had a major impact on software development. This past year we have seen rapid innovation and acceleration of digital transformation as a result of the pandemic. So, what does 2021 have in store for DevOps? How can DevOps teams improve productivity and become high performers in 2021?

    In this interview, CircleCI CEO Jim Rose joins Rebecca Auguste to talk about what 2021 has in store for DevOps and CI/CD, how the pandemic impacted software teams and what they can do to succeed in this new year.

    Jim is one of the distinguished speakers that will be sharing his knowledge and predictions for 2021 at this year’s Predict Virtual Summit. During his session, Jim will discuss what’s in store for engineering teams in a post-pandemic world, backed by anonymized team data from millions of DevOps workflows. Attendees will leave with insights and benchmarks to help teams build better software faster.

    Jim has led category-defining companies through economic crises, with COVID-19 being no exception. He is on a mission to provide everyone the ability to build and deliver software at the speed of imagination.

    To find out what the future of software looks like, attend Jim’s session at Predict Virtual Summit on Jan. 21. To check out the full agenda or to register for free, visit the Predict 2021 website.

  • DevOps for the Development and Delivery of High-Performance Applications

    DevOps for the Development and Delivery of High-Performance Applications

    Engineering teams that successfully implement DevOps practices often deliver software with higher business value. These organizations can easily identify areas where problematic code exists, quickly eliminate issues and deliver high-performance applications more quickly than their competitors that use traditional delivery methodologies. DevOps practitioners understand that creating an efficient model for performance testing goes hand-in-hand with four key objectives of digital transformation: extending performance testing to new roles; integration into continuous integration/continuous delivery (CI/CD) processes; end-to-end performance monitoring; and continuous optimization.

    Reduce Cycle Time and Time to Market

    In order to meet the requirements of rapid, high-quality application delivery, modern software teams need an approach that goes beyond conventional performance testing. This is where DevOps-centric approaches can provide faster, more efficient development and help ensure the delivery of high-quality software. Building the components of an application using agile approaches and validating features as they are created ensures that applications can be tested and verified more quickly, with no increase in associated costs or resources.

    By adopting DevOps practices, individual steps within the application delivery process can be automated as part of a DevOps tool-chain. For example, automating quality assurance test runs can eliminate – or significantly reduce – time-consuming human or analog searches for defects in code. Test automation can lead to an increased volume and expanded scope of testing, which, in turn, leads to a reduction in the defect escape rate. That saves resources, time and money.

    Shift Left to Increase Agility

    To mitigate performance errors and to address issues as early as possible in the application delivery cycle, tests should be executed during the build stage of the development process. By shifting left – running the tests as early as possible, ideally as part of a code commit – enterprises reduce the risk of discovering defects or issues with the product after significant amounts of development work have been completed. It is widely accepted that addressing defects later in development cycles takes significantly more time and money versus identifying and remediating defects early in the development cycle.

    As developers perform rapid, high quality, incremental changes, business analysts or product owners can more easily share changes, concepts or suggestions with their development teams. This continuous flow of communication can significantly reduce feedback time, increasing an organization’s ability to react to changing market requirements and allowing a level of flexibility within the development process that drives competitive advantage.

    DevOps in a Hybrid Landscape

    By supporting a wide range of tools and technologies, organizations that adopt DevOps practices can support applications that run in a local network environment, in cloud-based environments or, more commonly, in hybrid IT environments. As companies continue cloud migrations and transformations, well-designed DevOps tools can help simplify this transition while continuing to provide support for the legacy applications that provide core capabilities and processing for many enterprises. These hybrid tools are fully integrated into existing application development processes, allowing all application delivery teams to successfully transition to DevOps-centric practices without leaving any teams behind.

    One particularly important DevOps practice is performance optimization through continuous analysis. By using a centralized store and a common approach for quality-centric data collation, this practice helps analyze, enforce and improve the connections between software developer, CI Build and end-to-end performance testing to ensure a “quality first” mentality and higher-value delivery outcomes. The application of data visualization techniques allows teams to access code – build – test results in real-time and make decisions based on up-to-date insights.

    If DevOps practices are successfully established within a team or organization, they not only support more efficient software development and delivery but also are cost-effective. However, in order to successfully implement DevOps practices, enterprises need to understand and implement both technological and cultural changes. Through a combination of process, tooling and cultural simplification, and aligning teams toward common business objectives, organizations can deliver their end goal of creating high-quality, high-value software that satisfies the needs of the end user. By focusing on the common goal, and understanding the journey necessary to get there, enterprises can do so more quickly and at less cost – a win-win!

  • Set DevOps Free With Modern Application Security

    Set DevOps Free With Modern Application Security

    Development teams increasingly use technologies like cloud computing and microservices, in conjunction with DevOps principles, to innovate faster and remain competitive.

    But such progress comes at a cost. The fast pace of DevOps has left security teams scrambling to keep pace and install appropriate security guardrails. Traditional security practices used to occur in the final stages before release. The rapid release cycles of agile software development often make this difficult, if not impossible.

    The speed of development is outpacing the rate at which security teams can check configurations and scan for vulnerabilities, especially as developers now outnumber security professionals 500 to 1 and modern application environments represent a vast attack surface.

    The pressure to keep up, and do so securely, has led to a gradual reframing of DevOps as DevSecOps, where security is “shifted left” and introduced earlier in the software development life cycle.

    Unfortunately, while most businesses understand the intent of DevSecOps, they are still unsure how to translate that intent into action. In fact, just 14% fully integrate security throughout the software development life cycle.

    Recent research also shows that, while 65% of security teams report shifting left, less than one fifth are doing the scans necessary to verify that shift is actually happening. The same report suggests that most security teams don’t have the processes in place to monitor and protect cutting-edge application technologies, such as microservices, APIs and cloud native/serverless.

    This lack of visibility leaves security organizations blind when problems surface in production. The longer vulnerabilities go undiscovered in the development cycle, the more costly they are to fix. A study by IBM System Science Institute suggests that fixing a defect found during implementation can cost six times as much as one identified during early design phases. A defect uncovered in production can cost 100 times as much. Even more disturbing, nearly half of enterprises admit to deploying vulnerable applications, just to meet tight deadlines.

    It’s clear that compliance and security oversight are often overlooked, and even deliberately avoided, in favor of faster, more frequent deployments.

    The Great DevOps-DevSecOps Divide

    One of the most significant issues lies with developers’ ingrained perceptions. In the past, teams worked independently, with rigidly scheduled handoffs between distinct phases of development. Security operations (SecOps) teams often introduced security functions during the final stages of the development and release process, which resulted in delays.

    Successfully putting the “Sec” into DevSecOps depends on changing older cultural biases, reinforcing the need to embrace security and empowering teams with the right tools and automation to make smarter decisions without slowing down the entire organization. In essence, DevOps and security teams are all aiming for the same goal – a high-quality, timely product. The difference lies in how they measure and define “high quality” and “timely”.

    The truth is, development won’t be slowing down any time soon. Thirty eight percent of developers now release monthly or faster, and 54% of containers live for five minutes or less. Viewing security as a separate entity bolted onto the code, instead of a key feature that must be embedded end to end, slows down development processes and reduces efficiency.

    Delivering Speed and Security

    The core questions organizations must answer to improve application security are: What makes it easier for DevOps teams and application security teams to collaborate? And what does built-in security really look like?

    For organizations that want to improve their DevOps security, here are some tips:

    • Automate security as much as possible. Invest in security solutions that can be automated and embedded directly into the CI/CD pipeline. This makes it easier to secure applications without sacrificing development speed for the sake of security. Adopting technologies like static code analysis, dynamic analysis and penetration testing reduces risks and alerts developers to potential problems. There will never be enough security professionals to handle all security issues on their own, so use automation whenever possible.
    • Build security as a guardrail, not a gate. Provide appropriate guidance and tooling, so that security becomes a guardrail rather than a gate. For example, ensuring that DevOps teams have access to templated policies enables them to align applications with security requirements from the outset, without adding unnecessary time to development. Applications and security policies can also be tested as part of the CI/CD pipeline, so they are checked like any other functional specification. In short, it’s important to provide developers with everything they need to create and test applications with the appropriate security controls applied, or they won’t do it. Empowering development teams with a better understanding of security and self-service compliance reduces vulnerabilities and risks to the organization. Developers always want to move faster, so make it possible for them to “speed” safely.
    • Enable more proactive responsibility. The average developer cannot be expected to have the level of expertise necessary to stay current with all the latest security trends—they have enough trouble keeping their programming skills up to date. Reduce complexity and make it easier to gain developer buy-in by selecting solutions that provide simplified, easy-to-understand insights within the CI/CD feedback loop.
    • Make security adaptable, scalable and reliable. Secure applications with solutions that offer consistent, centralized and self-service security for any environment. For example, AI-driven security policy engines are one way to enable the adaptability necessary to support rapid change in applications resulting from CI/CD methodologies. Being able to adapt security policies in response to the latest attacks and identify dependencies makes it easier to assess risk and take action faster.

    The dawn of frictionless and adaptable security

    Security can no longer be an afterthought, bolted on to a process. Today, integrated security must become a normal part of any DevOps implementation. Making security frictionless and adaptable enables development teams to power ahead without fear. Modern application security can be a robust support system that empowers organizations to reach their business goals securely.

  • Improving Mainframe Agility With DevOps

    Improving Mainframe Agility With DevOps

    Mainframe workloads were on the rise even before the coronavirus pandemic, and transaction-processing mainframes became vital as lockdowns went into effect and every aspect of business and personal life occurred online. Mainframes are reliable, secure, highly available and affordable, and it is essential for organizations to ensure their mainframe can achieve peak agility to enhance profitability and improve customer experience.

    There are four areas of focus for DevOps teams who want to improve mainframe agility: teamwork, cloud services, KPIs and automation. Let’s look more closely at each one.

    Encourage Teamwork Between DevOps and Mainframe Ops Teams to Improve Agility

    It’s vital that mainframe ops teams work seamlessly with DevOps teams to ensure adoption of best practices in both areas, where practical, and that those best practices are adhered to. No matter how skilled your DevOps team, they must always be open to learning from mainframe professionals, operational and cybersecurity experts.

    To truly empower everyone on your team to achieve maximum mainframe agility, mainframe code must translate within the larger, multi-platform DevOps toolchain. If DevOps and mainframe code are not in sync, mainframe code could slow down software development lifecycles. Important processes, such as diagnostics, test execution and source code need to be integrated to enable DevOps teams to achieve better productivity.

    Enterprise DevOps is shifting focus away from software development exclusively to encompass surrounding processes within organizations. Successful DevOps teams must be able to work with cybersecurity experts and mainframe ops teams throughout the coding life cycle to ensure that everyone is on the same page and applications are functioning optimally.

    Decide Whether to Move Entirely to the Mainframe or Integrate Cloud Services with On-Premises Mainframes

    Scaling your organization for growth by migrating entirely to the cloud is not necessarily the right choice for everyone. In some cases, moving away from the mainframe’s core systems can be costly and unnecessary.

    Cloud-based services don’t have to be eliminated entirely, either. The cloud can and should be used for business processes that could benefit from real-time data updates, easy team collaboration and seamless integration with other software and tools. Mainframes excel at functions that require reliable, secure transactions.

    Whatever decision your organization makes, it must be communicated and agreed to at all levels. A customer-centric approach is ideal here, where the needs of the business are considered, but a seamless customer experience is prioritized.

    Establish Measurable KPIs

    Achievable and measurable key performance indicators (KPIs) are essential to monitoring your progress as an organization. KPIs for mainframe functionality usually include an availability rate of 99.999%, which roughly translates to downtime of less than 5.6 seconds per year. KPIs also generally measure time windows for batch completions. Software delivery velocity, efficiency and quality should be evaluated, but make sure you’re not prioritizing one of these metrics at the expense of another.

    There are more nuanced goals that need to be prioritized, too. Continuous improvements in software delivery are harder to quantify, but no less important. By fostering a collaborative environment where communication is key and teamwork is valued, your organization can create a better product or deliver better services for developers and customers.

    Use Automation Whenever Possible for Mainframe Agility

    Automated testing and AI-enabled processes can solve problems in real-time and free your DevOps teams to focus on larger issues. Mainframe ops teams can save their organizations money by relying on automated processes, which can also reduce personnel costs.

    Despite the benefits of automation, mainframe code is often removed from automated processes commonly applied to DevOps toolchains. Javascript, for instance, is one of the most commonly used programming languages due to its platform independence. It’s an essential tool for DevOps developers to enable the smooth configuration and installation of applications.

    But as infrastructures grow more complicated and surface areas expand to accommodate more users, operations teams can feel overwhelmed. With the continuous pressure to create new and better software on accelerated timelines, DevOps teams can unwittingly introduce vulnerabilities in Javascript coding as they speed through the software development and delivery pipeline.

    Automated mainframe diagnostics and optimization tools are vital to improving both the customer and developer experience.  Comprehensive and accurate data presented to mainframe ops teams in real-time can detect vulnerabilities and to find and fix potential issues.

    Ultimately, the mainframe is here to stay, with 93% of software experts citing the strength, reliability and scalability of the platform. Mainframes form the backbone of most monetized applications and websites, so it’s no surprise that mainframe agility has become a top focus for many IT departments. No matter how sleek and flashy your application is, if customers have a bad experience at the transactional level of the process, it could mean lost sales.

  • 7 Trends Influencing DevOps and DevSecOps Adoption

    7 Trends Influencing DevOps and DevSecOps Adoption

    Enterprise technology adoption trends can be broadly categorized under three pillars: infrastructure, architecture and technology. Let’s explore what these trends are and how they influence DevOps and DevSecOps adoption in tech corporations worldwide.

    Trends in Architecture

    In computing, architecture is the collection of protocols encompassing the utility, structure and execution of software applications. Architecture defines how an application works and determines the function of each component, such as data storage and computing capability, for example. Architecture trends directly influence how technology is developed, which also makes it influential on the field of DevOps.

    Trend #1: Monolithic to Microservice Architecture

    The shift from massive, inflexible systems that limit compatibility to lightweight, nimble software has increased DevOps and DevSecOps adoption. As architectures such as containers become mainstream, it is easier than ever for teams to code, debug and deploy software quickly, and work collaboratively. Microservice architecture offers the benefit of quickly being able to install, run and maintain systems, and is a catalyst to greater DevOps adoption, while containers spur DevSecOps adoption.

    Trend #2: Cloud-First Trend

    Mobile-first, cloud-first development has revolutionized data transportation, security and collaboration, and made development efficient, safe and transparent. This approach leads many organizations to adopt DevOps/DevSecOps.

    Trends in Infrastructure

    In IT, infrastructure refers to software, hardware, networking resources, systems and tools that allow companies to operate and manage their production processes.

    Infrastructure trends also heavily influence both the hardware and software tools companies use for tracking and manufacturing their products, and therefore directly impact their willingness to adopt DevOps and DevSecOps.

    Trend #3: Kubernetes Plus/Minus DevOps

    A significant infrastructure trend that affects DevOps is the increasing adoption of Kubernetes by global technology companies. Kubernetes is open-source software designed to orchestrate the management, deployment and scaling of containers, and  has seen widespread adoption in the tech industry. Kubernetes empowers organizations to be more productive, with an emphasis on security and collaboration. Containers in Kubernetes seamlessly allow multiple teams to develop a project simultaneously. Resource management and transparency are significantly enhanced, bugs can be fixed more quickly and work cycles can be shortened.

    Kubernetes and container management systems also reduce the need for human interaction and facilitate a fully automated, ‘NoOps’ scenario. Kubernetes uses a pipeline architecture approach that makes it easier for developers to use AI and ML tools to analyze, predict, and automate data and work processes.

    Furthermore, Kubernetes sets an infrastructure backed on logging, continuous detection, consistency, and self-healing, making it extremely easy and considerably faster for computers to make sense of the data, potentially eliminating human interaction. The infrastructure established by the trendy Kubernetes technology is both advancing and restraining DevOps adoption in companies worldwide, depending on how the companies implement Kubernetes into their work cycle.

    Trend #4: The Agile Infrastructure Trend

    Agile infrastructure for software development has become the standard for software development, and is perhaps the most influential trend impacting DevOps adoption. Agile, like DevOps, focuses on collaboration, organization, diversifying skill sets to achieve resilience, rapid iteration and self-improving work processes. Adapting to an Agile methodology means each team has members skilled in various disciplines that work in concert. Each product delivered is followed by a feedback cycle. Agile teams in an enterprise are transparent, collaborative and resilient to internal and external conditions.

    Trends in Technology

    Advancements in technology enable new ways of thinking and new best practices. As tech trends change, the work cycles, processes and mindsets also evolve.

    Trend #5: Release & Deployment Automation

    Manually moving software between testing and production environments is an inefficient, painful task that must be repeated every time updates are added, new versions are released or new features are built. But with the rise of release and deployment automation, software companies can entirely automate this process, saving time while boosting reliability. With the click of a button, test code can be automatically converted to production-ready work and deployed to version controls or hosting services. With this use of automation, DevOps adoption skyrockets.

    Trend #6: Site Reliability Engineers

    Site Reliability Engineers (SREs) are in huge demand as organizations strive to create highly scalable and secure software. SREs build robust, reliable software that leverages automation, create self-healing systems that focus on operations and align closely with business imperatives.

    Trend #7: Shift Toward AI/ML

    The evolution of artificial intelligence and machine learning is changing our entire perspective on the development and use of technology. With increasing automation, ML systems can process large amounts of data, make reliable predictions and provide extremely accurate analyses. AI can make use of this data and these predictions to determine what to do next, and create truly intelligent machines that can function with minimal human interaction.

    While this makes systems faster, more autonomous and reliable—which aligns with what DevOps promotes—it can eventually create a NoOps cycle that doesn’t require human intervention. In the short-term, this trend does influence DevOps adoption, but, in the long term, AI/ML could ultimately eliminate DevOps altogether.

  • How Healthcare Companies Can Optimize Software for Improved Patient Outcomes

    How Healthcare Companies Can Optimize Software for Improved Patient Outcomes

    Healthcare companies are more reliant than ever on software to help them achieve good patient outcomes. Recent research reveals that the use of software as a service (SaaS) solutions is growing exponentially in health care, covering a wide range of areas including:

    • Patient diagnosis
    • Medical imaging
    • Therapy and patient self-care
    • Data collection and analysis
    • Hospital management

    Due to the increased use of digital products in patient care, healthcare providers recognize the need to speed innovation and fundamentally rethink how they build software. This has led many engineering teams to move toward continuous integration and continuous delivery (CI/CD) when building and deploying software.

    Considering the industry’s emphasis on improving patient experiences, it makes sense that healthcare providers are keener than ever to optimize their systems. In fact, around 90% of healthcare providers are implementing or already using customized software solutions and development.

    As teams look to improve these critical processes in software development, feature flagging is one way to speed up development and optimize software performance in the healthcare industry.

    CI/CD Challenges for Healthcare Companies

    The healthcare industry faces unique challenges when applying DevOps strategies such as CI/CD.

    Intense Regulation

    The main problem is that, for obvious reasons, health care faces stringent regulation and oversight. A high level of scrutiny is necessary in an industry in which one small mistake or misjudgment can cost patient lives; however, in terms of DevOps, waiting for regulatory evaluation and approval can slow down the entire software development process, which defeats the core purpose of CI/CD.

    Security

    Data security poses another challenge for healthcare providers, especially when dealing with cloud-based software solutions. Cloud-based systems will always have potential vulnerabilities, as they rely on infrastructure that connects a variety of devices and smart technology equipment. There will always be inherent weaknesses in remote systems, as opposed to on-premises, too. A recent survey revealed that 42% of respondents said their major security concern was unauthorized access to cloud systems through insecure APIs. Network configuration problems were a close second, with 40% of respondents citing those as a concern.

    In a healthcare setting, another issue is the high volume and high velocity of data. Even a small, specialist clinic that may only see 150 patients per day will still generate vast amounts of data over the course of a month or year. Most of the data is private and sensitive, which means that data privacy and cybersecurity are top priorities for healthcare companies.

    The Impact on DevOps

    At first glance, because of the high level of regulatory oversight and cybersecurity concerns, it might seem that health care is a less-than-ideal environment for agile DevOps processes. How can you adopt fast-paced CI/CD cadences and innovate rapidly when there are necessary, opposing forces?

    The answer is to leverage innovative software development solutions such as on-premises feature flagging.

    Feature Flagging and Its Impact on Software Innovation

    Feature flags are, essentially, toggles or switches that allow software development teams to turn features on or off. The main advantage of feature flagging is that it allows for faster software deployment. One or more features can be coded into the software during development and made ready to go, but it is turned off for the user and/or patient on initial deployment.

    The ability to embed features within software that can be toggled on or off is perfectly suited for the healthcare industry since that approach allows for the necessary regulatory approval(s) before full deployment.

    For example, look at the rapid rise of telemedicine during the COVID-19 pandemic. Doctors, hospitals and patients alike were forced, along with the rest of the world, into the “new normal” of virtual interaction to mitigate the spread of disease while still providing quality treatment.

    For most industries, adding video chat/conferencing capabilities to existing applications was relatively easy. But for health care, the approval process, privacy and security concerns slowed down the adoption of these features. In our experience, typical healthcare teams are, best-case scenario, running on monthly—or, most likely, quarterly—deployment cycles to align their deployment efforts with compliance and governance review boards. Product and engineering teams work to get their updates and improvements in line with this cadence, which can make innovation difficult.

    If teams wanted to deliver this capability quickly, they could adopt feature flagging to decouple the concepts of “deploy,” “approve” and “release.” To use the video chat example, here are the steps they’d be able to take:

    1) Build the videoconferencing capability into their app and deploy the feature to production with the capability switched on for internal users (IP targeting, specific email addresses, etc.) only.

    2) Now that the feature is in live production, they can perform actual field testing to ensure that the functionality is working well, while simultaneously moving through internal and external approval processes.

    3) Once the review is complete, move to the release stage by switching the feature(s) on and allowing (external) end users to access the capability.

    All this can be achieved without needing to redeploy the software, and the developers are still free to make iterative improvements to the application.

    Now that the regulatory hurdles are addressed, it’s important to acknowledge that security is still an issue for healthcare companies that employ a SaaS feature flagging platform.

    While most providers focus solely on SaaS, the ability to deploy feature flagging software on proprietary cloud infrastructure is important in the healthcare space. This model enables both extra security and HIPAA compliance, and also opens up important capabilities and performance results that SaaS-only providers have a hard time matching. Since this technology typically becomes deeply embedded in the fabric of an organization’s product and technology, feature flagging is well-suited to an on-premises implementation.

    The implementation of on-premises feature flagging is a great way for healthcare companies to increase deployment velocity from monthly to daily. It allows their software development teams to adopt CI/CD cadences and innovate more rapidly, without the risk of breaching regulations or making sensitive data easily available to hackers.

  • Unlocking Your DevOps Automation Mindset

    Unlocking Your DevOps Automation Mindset

    In the new digital normal, where remote working has upended conventional software development cycles, the role of DevOps is central. Many enterprises have an accelerated digital agenda, but are often underinvested in rebooting DevOps for several reasons, including a lack of experience and insight into how it can help businesses recover from the new challenges created by the pandemic. This can be a strategic mistake, as real acceleration is driven at the developer level. Here are three ways DevOps automation can help enterprises overcome challenges in product life cycles and excel during this time of flux.

    DevOps Can Accelerate Product Life Cycles to Meet Demand

    In 2015, Adidas released a new line of Yeezy sneakers, which became an instant hit. The company quickly sold more than 2,600 shoes every five seconds. But Adidas’ IT infrastructure could not keep up with demand and crashed, and software developers were unable to find a quick fix.

    Impacted by customer and employee dissatisfaction, Adidas needed a solution to recover from the chaos and prevent future challenges. That’s when the company embraced cloud-native architecture, Kubernetes and DevOps. Today, Adidas’s IT infrastructure has accelerated its software product delivery cycle from once every six weeks to five times a day, while building a more resilient system.

    It is not that Adidas wasn’t aware of DevOps in 2015. The company hadn’t yet invested in educating its teams about the DevOps automation mindset, nor had it made DevOps automation a priority. Adidas needed a shift in culture, one that it brought about by rebooting its DevOps practice. This DevOps success story underscores how an accelerating go-to-market demand needs to be supported by underlying structural and systemic change.

    Today, as the pandemic continues to disrupt software development workflows, I urge enterprises to use this opportunity to reboot DevOps and gain speed, agility and resilience. DevOps can change the mechanics of dependencies between development and operations to reduce friction. By leveraging extreme programming and design thinking in conjunction with DevOps, enterprises can accelerate every segment of the software development life cycle independently, while maintaining resilience and adequate levels of governance. It provides the opportunity to invest in security-as-code (aka DevSecOps) so it is sustainable and scales easily. This also enables enterprises to reimagine software architecture and governance by implementing advanced digital architectures.

    For enterprises on the path to such transformation, there are four key DevOps Research and Assessments (DORA) metrics indicative of their DevOps practice’s success. These are: lead time for changes, deployment frequency, time to restore service and change failure rate. The first two of these could be seen as an “offensive strategy” and the latter two as a “defensive strategy” to measure speed. This means to accelerate delivery, organizations must introduce a faster lead time for changes to production, increase the frequency of new software deployment, shorten the time it takes to restore services to production and reduce the change failure rate to production. As you can see, Adidas needed improvements in all four of these areas to successfully reinvent its performance.

    DevOps Automation Can Reduce Complexities in the Software Development Process

    It took more than four years for BMW, the premium automobile manufacturer, to move from a waterfall software development methodology to a 100% agile, DevOps methodology. Today, 20,000 employees in the company use an agile software development toolchain, allowing them to streamline and synchronize their end-to-end software development. BMW has reduced its time-to-market by 25%, minimize incidents by 30% and gain a 360-degree view of IT.

    Today, enterprises are shifting from waterfall to agile weekly and daily releases. My belief is that every enterprise needs to adopt a 100% agile methodology, just like BMW did. Testing and continuous improvement/continuous development (CI/CD) is key for deploying code in small chunks and reducing merge issues and refactor efforts. Ultimately, this increases developer velocity and decreases lead time.

    The shift from a partial to a 100% agile model requires more than simply senior leadership’s resolve. It needs a dedicated pool of certified DevOps automation consultants, coaches and subject matter experts with experience in SAFE, LESS, Scrum and Kanban frameworks. Best-in-class enterprises and OSS toolchains that cater to DevSecOps, service meshes and omnichannel apps are essential. Simultaneously, agile-based delivery coaching, audits and continuous support to existing and new delivery teams are a must.

    While DORA metrics can serve as a good measure of an enterprise’s DevOps performance, businesses will need tools to assess DevOps maturity, improve developer productivity and provide specific recommendations for improvement. Data will play an important role in decision making and aid every developer’s performance, more than at any time in the past.

    Automation Makes Room for New Product Development

    Many software development teams struggle to keep up with the onslaught of new technologies entering the market. Take, for instance, artificial intelligence (AI); the number of applications built using AI is rapidly increasing. However, it continues to be new to many development teams. A similar story unfolds with Internet of Things (IoT), which creates complex systems with information pouring in from thousands of sensors and appliances every second with multiple software and hardware configurations. Testing these devices, configurations and performance cannot be done manually. This is where a DevOps automation mindset can help. It can ensure quality assurance processes have considerable test coverage across these different technologies and configurations by introducing automation.

    Newer technologies, such as Kubernetes, serverless apps, 5G, edge and cloud computing are increasingly used in product development, making DevOps even more important today. DevOps also plays a key role in transforming the developer experience. Today, developers need the foundation of agility, decision-making power and speed to deliver solutions on-demand. To facilitate this, enterprises must automate the management of development, testing, quality assurance and IT operations to successfully build continuous delivery processes. Further, given COVID-19 and the shift to working remotely, DevOps can help product teams collaborate remotely, leveraging well-established platforms available with any of the major cloud providers.

    As enterprises accelerate and adapt to work in this new normal, leveraging both the cloud and a remote workforce, a DevOps Automation mindset can speed up this transition. Enterprises must assess performance and enhance capability at the developer level to truly accelerate the journey to digital transformation.

  • Rapid Strategic SRE Assessments Accelerate IT Transformations

    Rapid Strategic SRE Assessments Accelerate IT Transformations

    A Strategic SRE Assessment is critical in helping organizations successfully transform traditional IT operations to SRE

    Site reliability engineering (SRE) is rapidly becoming more important to organizations that need to scale IT operations while maintaining or improving reliability and security, and to modernize operations to match the growth of DevOps in the organization. Without sufficiently thorough consideration given to best practices for people, processes and technologies, the transformation from traditional IT operations to SRE is unlikely to succeed. Despite challenges, the transformation to SRE is compelling because it provides an opportunity to reduce total operations costs and improve security and reliability while scaling services.

    To be successful, IT leaders need a comprehensive understanding of SRE best practices and need a well-engineered, yet affordable strategy and road map that effectively implements best practices and balances the many aspects needed to transform people, proces and technology. A Strategic SRE Assessment approach, based on best practices, is a key tool to help leaders understand, obtain the strategy and create a road map that will successfully transform traditional IT operations to SRE.

    There is no “standard” SRE assessment approach in the industry. This article enumerates a practical and proven Strategic SRE Assessment approach. The heart of the Strategic SRE Assessment approach is a survey of SRE best practices categorized under nine pillars: Culture, Work Sharing, Toil Reduction, SLAs/SLOs/SLIs, Measurements, Anti-Fragility, Deployments, Performance Management and Incidents Management. The survey is used to collect importance and capability scores for each of the SRE practices. As part of the assessment, a survey determines the current state and preferences of SRE practices for an organization for each of the nine SRE pillars.

    The following are examples of best practices for each pillar that are part of the Strategic SRE assessment survey:

    Culture – Leaders need to understand and sponsor a clear vision for SRE.

    • The culture embraces working practices to proactively shifts left the “Wisdom of Production.”
    • The culture encourages learning from failure and continuous improvement.
    • The SRE role is clearly defined and understood and SREs are considered a vital part of the organization.
    • Rules of engagement and principles for how SRE teams interact with their environment are codified—not only the production environment but also the product development teams, testing teams, users and so on. Those rules and work practices help to maintain SRE focus on engineering work as opposed to operations work.

    Work Sharing – Successful SRE requires that work be shared between SRE teams and development.

    • SRE teams are composed of software developers with strong operations knowledge or IT operations people with strong software development skills.
    • SRE workloads are managed to a budget with typically 50% or less spent on operations work.
    • SRE on-call work budgets are typically 25% or less. Developers share on-call work.
    • Technical debt is worked in small increments.

    Toil Reduction – SREs work to reduce non-value-added work through automation and standards.

    • Work standardization and automation—of both tools and processes—is employed to improve scalability, repeatability and other important goals.
    • Work standardization is used to reduce time spent on non-value-added work.
    • Automation is used to reduce time spent on non-value-added work.

    SLAs/SLOs/SLIs – Service level metrics, based on user perspectives, combined with error budget policies, are critical to managing flow and reliability.

    • SLOs are selected to measure availability, performance and other metrics in terms that matter to an end user.
    • An error budget (= 1-availability) internal service level objective (SLO) is established for each service. SLOs are set to be more stringent than external service level agreements (SLAs) promised to customers. The SLO budget considers what level users will accept, alternatives for users who are dissatisfied, and usage patterns of the product at different availability levels.
    • An error budget policy prescribes consequences if an error budget is spent. For example, the service freezes changes (except urgent security and bug fixes addressing any cause of the increased errors) until either the service has earned back room in the budget or the count period resets.

    Measurements – Smart monitoring implements observability to accelerate response times to incidents.

    • Monitoring solutions are designed to produce service level indicators (SLIs) from which SLOs can be calculated automatically.
    • SLIs do not require a human to interpret an alerting domain. Instead, software observability algorithms do the interpreting, and humans are notified only when they need to act.
    • Monitoring systems use telemetry and instrumentation to ensure the data being monitored is accurate, relevant and available in real-time.

    Anti-Fragility – Systems of people, processes and technologies are continuously tested and improved to assure they are resilient enough to serve applications as they scale.

    • Anti-fragility strategies are practiced to proactively assure the resilience of applications, infrastructures and pipeline services.
    • Fire drills are periodically conducted to proactively determine weaknesses in systems of people, processes and technologies that could potentially impact SLOs.
    • Chaos monkey or equivalent infrastructure testing strategies are used to proactively determine weaknesses in infrastructure systems that could potentially impact SLOs.
    • Security testing and DevSecOps practices reduce the risk of security vulnerabilities.

    Deployments – Deployment strategies reduce the risk of deployment failures.

    • Non-emergency deployments use progressive rollouts in which changes are applied to small fractions of traffic and capacity at one time. If unexpected behavior is detected, the changes are rolled back first and diagnosed afterward to minimize mean time to recovery.
    • Automation is used to implementing progressive rollouts, detect problems and implement rollback changes safely when problems arise, because outages are primarily caused by changes in a live system.
    • Deployment strategies such as blue-green, feature flag rollouts, A/B testing and canary rollouts are used to reduce the blast radius of failed deployments.

    Performance Monitoring – Proactive testing and monitoring ensure applications and infrastructures have the capacity necessary to flex and scale when needed.

    • The SRE team is responsible for provisioning and monitoring utilization because utilization is a function of how a given service works and how it is provisioned.
    • The SRE team is responsible for capacity planning and provisioning. Regular load testing of the system is used to correlate raw capacity (servers, disks and so on) to service capacity.
    • Provision to handle a simultaneous planned and unplanned outage without making the user experience unacceptable. This results in an “N + 2” configuration, in which peak traffic can be handled by N instances (possibly in degraded mode) while two instances are unavailable.

    Incident Management – Incidents are managed in an efficient, blameless manner with a goal to use incidents to improve responses.

    • Playbooks record the best practices for human response actions because playbooks produce roughly a 3x improvement in mean time to recovery.
    • Emergency response performance is measured using mean time to repair (MTTR) metrics instead of mean time to failure (MTTF) because the most relevant metric in evaluating the effectiveness of emergency response is how quickly the response team can bring the system back to health—that is, the MTTR.
    • Postmortems are blameless and focus on process and technology, not people.
    • Postmortems are conducted for significant incidents regardless of whether they resulted in a page; postmortems that did not trigger a page are even more valuable, as they likely point to clear monitoring gaps. Find all root causes of the event and assign actions to correct the problem or improve how it is addressed next time.

    SRE Assessment Surveys Are Not Enough

    The survey alone is an essential tool to collect information about the current state and preferences for each of the SRE practices. However, this is only part of the comprehensive Strategic SRE Assessment approach.

    The Strategic SRE Assessment approach starts with workshops to get alignment between leaders and teams regarding big-picture goals prior to conducting the survey. This is necessary so that everyone who answers the survey has a common perspective and can better respond with considered views of importance and capability scores for each of the practices. The results of the survey include an analysis that points out specific pillars and individual practices that have a high gap score in which the practices are scored as important but not yet practiced well. The high-priority gaps are then used to organize a value-stream map working session, that is designed to find the highest priority bottlenecks in people, processes and technology bottlenecks in the value steam, relative to the organization’s goals and gaps. With the goals, gaps, and data from the value stream map in hand, the next step is to formulate an implementation road map that directly addresses the goals, gaps and bottlenecks in the value stream.

    The final step is to obtain strategic alignment of the big-picture plan by reviewing the findings and road map with the leadership and implementation teams. This entire Strategic SRE Assessment approach, from goal alignment through surveys, value stream mapping, road map creation and strategic alignment, varies depending on several factors such as the size of the organization, number of applications and size of the deployment domains to be included in the assessment.

    A Rapid Strategic SRE Assessment for a sample set of model applications typically is conducted in 21 days. More complex Strategic SRE assessments involving many applications may take up to 90 days.

    In most cases the Rapid Strategic SRE Assessment is the best place to start. Most often, it is not a good idea and is too expensive to “boil the ocean” and make changes across all applications and deployment domains at once. By starting with a small number of model applications, it is easier to implement, measure and demonstrate success and use the results at each step to justify ongoing investments needed to complete the transformation across the organization. Another advantage of the Rapid Strategic SRE Assessment is that it is a quick and affordable approach to check-point progress as SRE is implemented progressively across applications and deployment domains.

    Key Takeaways for SRE Transformation Leaders

    SRE offers immense value for modernizing and scaling IT operations reliably and securely and to complement DevOps transformation. Adherence to SRE best practices is essential to reduce risk and assure security as the transformation is implemented.

    Each organization is different and has different requirements and priorities for its applications and deployment domains. The specific SRE practices that are the highest priority vary according to the priorities of applications and deployment domains. The Rapid Strategic SRE assessment approach, based on nine pillars of SRE best practices, is quick, effective and affordable.

    The Rapid Strategic SRE Assessment is a preferred approach to kick-start the SRE transformation. This approach is also a preferred and affordable tool to reassess the direction as SRE is implemented progressively across the organization.

    Summary

    There is no “standard” SRE assessment approach in the industry. The Rapid Strategic SRE Assessment approach described in this article has been tried and proven with large and smaller organizations. The author invites your comments regarding this approach and the SRE best practices.