Tag: Chris Corriere

  • The DevOps Equation: Agility + Empathy = Quality

    The DevOps Equation: Agility + Empathy = Quality

    I had the chance to catch up with Jeff Sussna (@JeffSussna), founder and principal of Ingineering.IT, ahead of his keynote address on continuous design, scheduled for DevOps Days Atlanta, April 26-27.  He discussed the importance of designing for service, responding to the unexpected, and the importance of building empathy across teams.  

    Derek Weeks (@weekstweets): Today I’m really happy that we have Jeff Sussna joining us for the latest in this series. Jeff, why don’t you introduce yourself?

    Sussna: Sure. Thanks for having me. I’m an independent consultant. I’ve been around, since rocks were young or the 80s, whichever is older. My particular background is that I have built systems and led organizations across the entire development, QA and operations spectrum. So, I’m a little unique in that respect.

    My current focus is on helping IT organizations and digital businesses really bring together their ops and design thinking so they can not only go faster and not only be more resilient, but provide more customer value at the same time.

    Weeks: Let’s talk about that a little. How is continuous design—a subject you have written a lot about—coming into play? How can it help us be more resilient.

    Sussna: Just to explain a little about what we mean by continuous design. It’s basically bringing together the entire organization and a unifying design in operations. It’s taking what we are doing in DevOps and expanding it to include product, design and support. It lets the whole organization really have an agile conversation with a market and with its customers.

    The issue that we face is that we live in a service world now. The point is no longer delivering products like cars coming off an assembly line. It’s much more about responding to our customers and having relationships with them. This is happening in a world that is more and more complex while less and less is in your control. We can’t figure everything out in advance. We have to have the ability to respond to the unexpected. We have to be able to do that at every level.


    “This is happening in a world that is more and more complex, while less and less is in your control.”


    I will give you a couple of examples of things that are perfect illustrations of this:

    I recently wished a friend of mine happy birthday. The reason I did that was that because there was a notice on my calendar that said it was her birthday. It was wrong. It wasn’t her birthday. I had no idea where that notice came from. Did it come from Skype? Did it come from LinkedIn? Did it come from email? I had no idea.

    We face that in IT, too. Systems are becoming more complex. We develop microservices. We use public and private clouds at multiple levels. We’re no longer in a situation where we can control things. We have to be able to respond and respond at all levels.


    When we make design and operations unified and continuous, we are able to say, “Where did we think we were going? Where would we need to go? and what do we do next?”


    What I talk about in my book, in my workshops, and all my work is that this needs to happen at every level. It’s not just a matter of designing for failure inside the data center or inside the infrastructure. We also have to do it at other levels.

    Here is another example:

    I was trying to get paid recently by a large enterprise client in Europe. They were trying to wire money internationally and my bank wasn’t really set up for it and they were having problems. They just kind of gave up. I kept getting these messages saying, “Sorry, we tried paying you and it didn’t work.” That was the end of it.

    I started to get frustrated and my client within the enterprise started to get frustrated to the point where they sent a scathing email to our AP department which they CC’d me on, saying, “Why can’t we take the lead to get his guy paid?”

    That’s an approach that is based on resilience in the sense of assuming that something can go wrong and figuring out what he can do to fix it when it can go wrong. As opposed to the traditional approach where we’ve been trained to push certain buttons and take certain actions. If those actions don’t work the way they are supposed to, well, I guess that is someone else’s problem.

     

    https://www.youtube.com/watch?v=ISSMHmwIMtA

     

    Weeks: This brings to light some recent conversations I was having with Paula Thrasher at CSRA. She was talking about developing systems and how to make them more resilient. She is using pictures as part of her conversations with people to help visual their work. What work steps are there? What work is flowing? What is communicating with what?

    Are you taking similar approaches with continuous design in how you work with clients to pursue it?

    Sussna: Well, I try to introduce them to the principles and I also try to introduce them to a set of practices which actually comes out of Mark Burgess‘s work on promise theory. It is really a conceptual model for how we think about success and failure and how do we think about service. For example, Amazon helps you get everything you need in your life from shaving cream to batteries to snow tires to books as quickly and easily as possible. In order to keep that promise they make a whole variety of other promises. Some of which filter all the way down to a promise to be able to go over and make a change to production as quickly as possible.

    Once you take that mindset. The interesting thing about a promise and the reason we use that word is because it is very intuitive. We all know what a promise is. We all make promises to each other every day. On the other hand, when we think about it in terms of service, IT, or whatever it may seem a little odd. The thing about promises is that on the one hand we are making commitments to do things and on the other hand promises sometimes get broken. That is actually good because it forces us to think. If I promised to pay my vendor. What do i do if I’m not succeeding in paying my vendor?


    The thing about promises is that on the one hand we are making commitments to do things and on the other hand promises sometimes get broken. That is actually good because it forces us to think.


    It’s sort of flipping our mind. Again, we are doing this in DevOps world where you start thinking about things like mean time to repair (MTTR) instead of mean time between failure (MTBF). We shift from how do we create a situation to how do we create a situation from where nothing will ever break—which becomes less and less possible. When then need to shift to a situation where we can respond more and more quickly.

    We go to executives and say, “We don’t want to worry about how often stuff breaks anymore,” and they freak out. We have to teach them and we have to teach ourselves that we can’t prevent things from breaking. Instead, let’s be able to fix them so quickly that maybe no one notices. My focus is on really helping people shift their mindset from thinking about ourselves to thinking about who we are helping. And from thinking about what we are doing to thinking about how well it succeeds and how well it overcomes problems.


    We have to teach them and we have to teach ourselves that we can’t prevent things from breaking. Instead let’s be able to fix them so quickly that maybe no one notices.


    Weeks: You talked about delivering on promises as quickly as possible. How does continuous design account for doing things as quickly as possible but making sure the quality is there at the same time?

    Sussna: In two ways—one way is that when you do things in small increments you are able to find problems more quickly. This is classic; it comes out of lean. It comes out of integration. It’s pretty basic stuff for our community now that it’s much easier to find and fix a bug five minutes after you change a code than five weeks after you change the code.

    When you put things like Linux, agile, DevOps, and design all together, they all focus on incremental feedback loops where you discover the gap between what you think you’re accomplishing and what you’re actually accomplishing very quickly. Secondly, when you take this all to the way to the marketplace and you make it part of the relationship with your customer, you can reduce a tremendous amount of waste.

    The way to think about continuous design is: it’s all about steering. Think about a boat and you’re trying to sail across a bay. You can’t just point the boat at the other side and say go because the winds are changing and you’re going to be able to respond to things that you can’t predict. If you sail for 20 minutes then discover you had pointed in the wrong direction, you’re going to be way off-course. You’re then going to generate a tremendous amount of energy getting back on course.

    What continuous design says is just an extension of continuous integration and continuous delivery taken all the way though the business. When we steer continuously we steer efficiently.


    What continuous design is really just an extension of continuous integration and continuous delivery taken all the way though the business. When we steer continuously we steer efficiently.


    Weeks: I know you’re going to be keynoting DevOps Days Atlanta coming up in April. Chris Corriere, one of the guys on the organizing committee, tells me that you’re leading a full day workshop on continuous design before that. Maybe you could tell some of the people that are attending DevOps Days Atlanta why they should be there a day earlier for your workshop and what things you’re going to address during that time.

    Sussna: To some degree what I am going to address is everything we have been talking about for the last 10 minutes or so—relative to DevOps. I think the key thing is, and we need to think about it. We need to think about more than just going fast. One of the big misunderstandings that I see with DevOps is this notion that it is primarily about delivering application changes to production faster. That’s only half of it. We really need to think about from the perspective of providing service.


     One of the big misunderstandings that I see with DevOps is this notion that it is primarily about delivering application changes to production faster. That’s only half of it. We really need to think about from the perspective of providing service.


    You know what we are talking about is essentially developers and operations people providing better service to each other. As a result, they can deliver better service to the business.

    Where we talk about microservices. You know that word service is in there for a reason. If all you do is you kind of shatter your organization and shatter your architecture into a bunch of little pieces, you still have to figure out how those pieces come back together into something coherent. The answer to that is taking a really deep service-oriented approach. That is very much what my workshop is about. What does that word even mean and how do we do it?

    Weeks: You mentioned thinking about interactions as delivering better service to one another within the organization. How does empathy come into your thinking around DevOps and Continuous Design?

    Sussna: I’d like to say agility plus empathy equals quality. What I mean by that, in particularly empathy, I think it’s important to take a minute and clarify. When we talk about empathy we are not talking about wallowing in someone else’s pain. We’re not even necessarily talking about things on an emotion or feeling level. It’s very simply the ability to see things as if from an other’s perspective. I will give you a prime example of where that comes into play and how it’s important and how it helps.


    Agility + Empathy = Quality


    When I work with clients, I tend to look at their process using their organization from the product beginning to the support end. Often times I see support getting the short end of the stick. I see places where they get phone calls from customers with features that they don’t even know exist yet. They sometimes get talked down to by development or product or even operations. They say things like, “Well you’re just supposed to support the stuff. You’re the young people. You’re the less experienced people.”

    When we have the ability to see things from their perspective, we can think about how support is actually experiencing our delivery process. If we don’t think about them when we go through the process of delivering new features, they will be lost. Them being lost will show when customers call for help. That will significantly affect customer satisfaction and then even beyond.

    The reason we need to empathize with support is so we can empathize with the customer. Often times when we build things and we don’t think about the notion that our customers might have trouble with it. On the one hand we have the illusion that if we design it and we build it really well, they won’t have trouble. We can’t perfectly predict who they are, what they need or what they will encounter. There’s an inevitability to the idea that they are going to need some amount of help.

    Weeks: I think that is probably the best description of empathy within a DevOps environment that I have heard anyone talk about. Jeff it has been a great conversation. Thank you very much. I really appreciate it and we look forward to seeing you at DevOps Days Atlanta coming up.

    Sussna: Thanks. It has been great talking to you and I am really looking forward to the conference. It will be fun.

    If you want to learn more about continuous design from Jeff, be sure to join us at DevOps Days Atlanta. I will also be at the event giving an Ignite Talk, titled, “Real Heroes Draw Pictures,” focusing on lessons learned from the 31 DevOps and continuous delivery reference architectures I assembled.

  • Rugged DevOps: Less Capture the Flag, More Teamwork

    Rugged DevOps: Less Capture the Flag, More Teamwork

    At the recent DEVNEXUS conference in Atlanta, I caught up with Chris Corriere to talk about his experiences in the realm of Rugged DevOps. Chris is a DevOps Engineer at AutoTrader and a contributor here at devopsy.kinsta.cloud. During our conversation, we discussed automation, culture and collaboration, and which thought leaders he is following. Chris also shared insights on the upcoming DevOps Days Atlanta conference where he serves on the organizing committee.

    Derek Weeks: What’s your definition of rugged DevOps?

    Corriere: Rugged DevOps to me is definitely manifested in the security sector but it’s a shifting quality and awareness left in doing the best we can with the tools we have, the constraints we’re given, and in the environment we’re in. It’s about being more experimental, understanding that we’re going to encounter failure, and learning to adapt accordingly. We’re very much having to make tough decisions in real time. If you’re running anything in production these days, you need to do that as safely and as consistently as you can.

    Weeks: Is rugged DevOps only about security?

    Corriere: No. Security is really about mitigating risks. I think a lot of this ties into situational awareness about understanding your company’s risk. Is it user data? Is it credit card transactions or do you have a Russian botnet running in your infrastructure that you’re not aware of? What’s the asset you need to protect? Are you trying to protect an asset that is not there?

    You can’t have security come in and become suffocating where it’s not needed. In other situations you can’t leave things wide open that really do need to be protected because you’re more concerned with moving quickly than moving safely.

    https://youtu.be/fgMPKkM1NdY

    At the end of the day when you’re mitigating risk, it doesn’t matter if it’s customer data or a broken build or a bad user experience, these are all risks we need to mitigate. That’s really what rugged DevOps is. It is about is tying that value back to the user. It’s knowing what problems we trying to solve and understanding what we doing effectively (or not). It can be locked down as tight as you please, but if you cannot use the tool to solve the problem you picked it up for, it might as well not be running.

    Weeks: Security teams have been one of the last parts of the organization to come into DevOps practices. What’s been your experience with engaging security teams at Auto Trader?

    Corriere: That’s a good question. I’ve had experience with more than just security teams at Auto Trader. I’ve been involved in security a few places and they’ve approached me first when they found out I was into the DevOps stuff. They often come looking for assistance. They want to know: How do we get this baked in? How do we make this a priority?

    It really ties back into multi-factor authentication. It’s a good way to frame it up where you want developers and operations (those people who aren’t permanently rooted in security) to be aware of these things, to have automated scans, to be aware of cross-site scripting, and to understand how to remediate those kinds of issues.

    We’ve been implementing open source tools earlier in the development lifecycle to give us quicker feedback loops in order to be able to triage these things. Security applauds that but they can’t trust us too much with it. They’ve got to verify that we’re doing a good job with this, right? It’s a give and take where I’ve seen security come and offer automated solutions to DevOps teams and those solutions get adopted. The DevOps teams then take that and really start to run with it. Then security wanted to slow down a little bit because they were concerned about the quality of the vulnerability remediation. There is some back and forth there that has to be anticipated.

    ________________________________

    “We’ve been implementing open source tools earlier in the development lifecycle to give us quicker feedback loops.”

    ________________________________

    For security to trust you you’ve got to trust security to a large extent. If you’re putting more automation into the left side of your process and doing things before they hit production should be making your job easier. One of the side effects of that security may start going through some systems with a finer tooth comb because they’re not doing as much fire fighting. That may result in another ask from security. Understanding that this isn’t a declaration of war—we are simply continuing with an incremental progression toward further improvements. Interactions are cyclic and will happen again.

    We need to help security understand our pipeline, determine what’s going to be a good fit, and keep that communication level high so the trust is there. In these symbiotic relationships, cooperation is dependent on that trust. If that trust goes away you end up in something more competitive.

    There’s always room to drop back to where it’s more commensal and knowing when the kitchen’s gotten too hot. At those times, everybody needs to take a cool down lap and come back to it at the beginning of the next month. It goes back to that situation awareness, knowing where you’re at right now before you try to get to where you’re going.

    Weeks: We talked earlier today about red teams, blue teams, purple teams. What are they and can you describe some of the practices you’ve used to end up with more purple teams?

    Corriere: Yeah. Red teams take an attacking, reactive posture. Red security teams are worried about internal compromise, and when they find gray areas—for example, things that haven’t been fully signed off and authorized—it can result in unplanned work cropping up for the DevOps team. Generally that kind of unplanned work is introduced because the “Red” team felt they were not able to accomplish their job another way.

    Then the blue team side of the equation is really trying to stomp all that out. Blue teams keep things standard and locked down. They make sure everything’s approved in triplicate before it gets implemented, which can be rigid and very frustrating for developers.

    Purple teams allow that experimentation in earlier in the development lifecycle where they’ve got a little bit more leash. They’re allowed to experiment and stand up new things and make those things work before they getting it approved. Again, it’s about being transparent. Teams first want to understand what change was introduced and what the advantages of it are. They may also acknowledge that you need more than one solution to some problems, and that one tool might not fit everybody. If they’re using tools to automate that’s good. We can’t box them all into one necessarily. That’s where you see this go from blue and red to more of a purple. There’s more communication. There’s more conversation. There’s less capture the flag, more teamwork.

    ________________________________

    “There’s more communication. There’s more conversation. There’s less capture the flag, more teamwork.”

    ________________________________

    Weeks: You’ve been involved in the conversations around rugged DevOps for awhile. Who are some of the other thought leaders you learn from?

    Corriere: Jamesha Fisher out of San Francisco is one. She’s done some neat security stuff and is starting to look more into the vulnerabilities around some of the automation tools. Where are those vulnerable and where can they be leveraged against us? Automation is great until it automatically starts doing things we didn’t want. Another aspect of that is really trying to think more like an attacker; we need to think about: how would I break into this thing instead of just thinking how do I protect it.

    Georgia Weidman is popular in a lot of security circles and I’ve missed her workshop once. I’d like to catch it. She’s got a book out on pen testing which covers a lot of operations and concepts from an attack perspective. I would say that can be a fault of some rugged DevOps practices; you’re so quick to get the thing working and provide business value that you don’t realize where you’ve left something vulnerable. Sometimes you’ve got questions about why that thing you work with frequently is always locked down. Perhaps you find you can’t use it the way you want to. At those moments, it’s time to read through Georgia’s book. You’ll get some information as to why that thing is not open the way you want and where someone might use it for reasons might not expect.

    ________________________________

    “You’re so quick to get the thing working to provide business value that you don’t realize where you’ve left something vulnerable.”

    ________________________________

    Weeks: As we wrap up I know you’re on the organizing committee for DevOpsDays Atlanta. Do you want to give us an insight on the dates of when that’s coming up and call for papers, et cetera or we can help you share that?

    Corriere: Our CFP is open through the first of March. Our event is going to be in April. It’s the 26th and 27th which is a Tuesday and Wednesday. Monday, the 25th of April, we have Jeff Sussna coming in to do a full day workshop on continuous design. He’s going to be giving the keynote on our first day. We’re excited to have Jeff in town and excited to have him as a keynote. John Willis is going to be giving our keynote on the second day.

    We’re looking for proposals from anything in progress, people learning in the field, or trying to drop that barrier of entry. We’ve got a lot of bright people in the space. We’ve got a lot of people learning as they go. We need to be more vocal about that learning process and how we find our way through it. There’s not always this instantaneous vision that strikes us. We understand how to convert a shop into continuous delivery every night. There are definitely steps in between that. We need to have more conversations about those step in our industry. Yeah, anywhere in the stack, any interesting projects going on, we’d love to hear from you if you’ve got something.

    If you are interested in learning more about this subject, I invite you to download Amy DeMartine’s Forrester research paper, “The 7 Habits of Rugged DevOps.”

    As Amy notes, “DevOps practices can only increase speed and quality up to a point without security and risk (S&R) pros’ expertise. Old application security practices hinder speedy releases, and security vulnerabilities represent defects that can leave a company open to cyberattacks. But DevOps practitioners can leap forward with both increased speed and quality by including S&R pros in DevOps feedback loops and including security practices in the automated life cycle. These new practices are called Rugged DevOps.”

  • DevOps Leadership Series: Security at Velocity

    DevOps Leadership Series: Security at Velocity

    If it does not fit, it does not get done.  For many DevOps practices, application security falls into the “does not get done” bucket.  That’s because for many DevOps-centric organizations, application security has historically been done somewhere else, by someone else, who is slow.

    Go faster.  Shift left.  Remove complexity. Reduce rework.  All mantras of DevOps practices.  And while DevOps practices have changed dramatically in recent years, many experts will tell you that application security has not changed enough.

    In this installment of the DevOps Leadership Series, you will hear Chris Corriere (DevOps Engineer, Autotrader) and Mitchell Ashley (VP Information Technology, CableLabs) share perspectives on the state of DevOps and security. It is about security at velocity.

    But is going faster and automating resulting in better security? There are still security pros who are uncomfortable with that premise. Nevertheless, that is where this is headed. The security community has to come to the mountain, otherwise the mountain is coming to them.

    First, listen to Chris’ Corriere of Autotrader tell us his perspective that security can move at DevOps speed, as long as you take a diversified approach:

    https://youtu.be/8DdfHkAK1Fk

     

    Then hear from Mitchell Ashley, VP of IT at CableLabs as he remarks on the evolution of security from a validation role (at the right) to an integrated, ingrained role that has shifted left:

    https://www.youtube.com/watch?v=pg_m7dYqUPU

    This is the third in our series of interviews from this event. Stay tuned as the actual sessions from DevOps Connect will be posted shortly.

    NOTE: If you have missed any of the other videos from this series, you can find them here.  (We’re up to 15 so far).