devopsy.kinsta.cloud

Tag: software development

  • You’re Thinking Software Wrong

    You’re Thinking Software Wrong

    Like any industry, DevOps fosters its own unique culture. And like any culture it resists change, it has inertia. “This is how it has always been done” is an incredible force to reckon with.

    Even though the realities of DevOps have changed dramatically in recent years with the advent of new tools and services, many ideas that have been ingrained in the culture will probably still be with us for ages. Some of these ideas are not only wrong, they border on being dangerous. One such fallacy is the sacred rule cherished by development teams and system administrators: “NEVER deploy on a Friday afternoon.” That is, unless you want to spend your entire weekend fixing bugs. A more pernicious example is the division of labor between us, humans with “root access” to machines, and the robots, the tools teams create to make their processes less manual. These days you’ll still find many development groups that delegate the menial tasks to automation, but keep the most dangerous, infrequent tasks to human operators. These are two prime examples of “thinking software wrong.”

    These ideas are not only incorrect, they can be crippling to an organization, causing delays in software development and leaving the company vulnerable to data breaches, for no good reason. These fallacies are what we refer to as “broken culture”—what happens when culture is deeply misaligned with the interests of the organization, the individuals who compose it and its stakeholders. It usually manifests when historic cultural norms prevent members from embracing new and better ways of doing things.

    Fear of Friday deployments is our first example of broken culture. This idea has taken a strong hold in DevOps culture because developers haven’t trusted their own software and systems enough, and they don’t want to spend their Friday nights (or the whole weekend) debugging a production problem. Fair enough, but there is a better way.

    Let’s look at another example: the Equifax breach. This incident—which cost the company billions of dollars and continues to haunt consumers to this day—happened because a certain procedure mandated that a specific piece of software must always be updated by a human. Since humans are prone to error, this lone simple procedure (designed and supported by years of a broken DevOps culture) resulted in a devastating data breach. Once again, there is a better way.

    Deploying is not the same activity it was a decade ago. Advanced automation tools are drastically streamlining deployments, significantly reducing the chance of error and speeding the time it takes to identify and correct any errors that do occur. This means companies can deploy any day of the week and any time of the day, worry free—and can let the deployment date be dictated by what is best for the company and its customers, not the weekend plans of developers.

    The same concept applies when it comes to updating software. A better way is to leverage the automation tools that are now readily available, and allow organizations to have robust management of compliance requirements without human error or delay.

    In addition to embracing better tools, here are a few other ways companies and developers can rethink software for the better:

    • Software must be valued as more than just the cost it takes to create it, or to replace it. Code should be an asset, not a liability.
    • Software should be thought of as constantly changing—a dynamic system with which people, machines and processes interact continuously. When this never-ending process is supported by the right tools, deployments are easier and managing and updating software is seamless and secure.
    • Code by itself is useless. It needs infrastructure to build it, and data to create something meaningful. Therefore, if you only test the code, but are not managing your infrastructure in the same way, you’re not really testing the software system.
    • Companies can dictate the cadence at which they create new software, but they do not have this same luxury when it comes to fixing broken software. Identifying and fixing problems must be done as soon as possible.
    • Every new update or fix must be version controlled so that everything is auditable. That includes changes to infrastructure configurations!

    Once companies truly understand these fundamental truths about software and embrace the modern tools available to them, they can start to shed the aspects of DevOps culture that are limiting or damaging.

    Companies can deploy new apps and features on Fridays, confidently and without issue. Companies can even rebuild an entire infrastructure from scratch, in minutes, while applying all security updates for every single service and library—and be completely sure nothing will break. If you can’t do that—or don’t believe you can—you’re thinking software wrong.

    — Ori Pekelman

  • DevOps Chat: disrupt:Ops Brings Security Closer to DevOps

    DevOps Chat: disrupt:Ops Brings Security Closer to DevOps

    Shifting security left during the software development process can be tough for companies because of the sheer number of operational things that go along with implementing security that must occur somehow in the realm of DevOps. It can be easy to lose control over their security processes without some sort of guardrails in place to help.

    A brand new company is helping do just that. I’m excited to introduce disrupt:Ops, which bills itself as a guardrails platform to help companies include and enforce best practices across security and operations, bringing security into the DevOps process at the beginning. My friend Mike Rothman, disrupt:Ops co-founder and president, joins me in this DevOps Chat to discuss the vision of his company and how it hopes to help make security truly a part of DevOps.

    You can read more about disrupt:Ops and their guardrail approach to cloud security and devsecops from another co-founder of the company, my friend Rich Mogull at https://devopsy.kinsta.cloud/building-great-cloud-security-guardrails

    As usual, the streaming audio is immediately below, followed by the transcript of our conversation.

    Audio

    Transcript

    Alan Shimel: Hey everyone, it’s Alan Shimel, devopsy.kinsta.cloud, and you’re listening to another DevOps Chat. Breaking some big news today on DevOps Chat, the launch of a labor of love of some friends of mine. I’m happy to be joined by Mike Rothman, cofounder and president of disrupt:Ops. Mike, welcome.

    Mike Rothman: Alan, it’s great to be here, thanks. It’s always fun to catch up with old friends.

    Shimel: Yep, and old friends it is, Mike. We should let our audience know – maybe we shouldn’t let them know. But we are old friends. We go back a long time now, back to the beginning of blogging and security –

    Rothman: That’s right.

    Shimel: It’s got to be 2004, 2003 or something like that.

    Rothman: Yeah, that’s right, that’s right. When I had dark hair. Actually I didn’t have dark hair back then, but you know, it’s –

    Shimel: I had hair.

    Rothman: You had hair. That’s right.

    Shimel: I had hair. But you know what, Mike? I said it was a labor of love and everything else. Just in the way of background, why don’t you share with our audience your personal sort of journey in tech and, you know, work history.

    Rothman: Well sure. I started out, God, in what feels like 100 years ago, as a mainframe programmer. Ended up doing some local area networking stuff. Kind of found my way to network security. Went into research for quite a while. Did a couple of companies and then went back to research and for the last 10 or 11 years at this point I’ve been partnered up with Rich Mogull and Adrian Lane at Securosis, doing a bunch of kind of information, security-centric research. And actually about a year and a half ago we spun out some technology that we had prototyped within Securosis in cloud management and cloud automation into our new company disrupt:Ops, and have been just making a lot of progress over the last year with that. And that’s what we’re announcing this week.

    Shimel: Yep. So Mike, you’re being humble. When you say you did research, you were an analyst at Giga, were you not?

    Rothman: META Group, yes. META Group, exactly.

    Shimel: META Group. And then you were also – you had executive positions in several cyber – we didn’t call it cyber – infosec companies back then. 

    Rothman: It was. Yeah, you bet. So companies like TrueSecure and CypherTrust, you know, back early in the e-mail security days. I did a network monitoring company called EIQ. So yeah, I’ve been there, done that. Have the road rash and kind of the screw-ups under my belt to prove it. But you know, it’s interesting in terms of just how long and how small an industry security is for being actually a really big thing nowadays. 

    Shimel: Being so big, yeah. It really is. And Mike, again, for many people in our audience who come from DevOps or even from security – but they’re worldwide and they not be familiar with Securosis. You know, Rich Mogull comes over from Gartner. Adrian’s been CTO at major companies. You know, you guys are probably the leading, let’s call it independent security analyst research firm – you said 10 years-plus. Certainly in my time you guys laid the fundamental kind of – a lot of the fundamental backbone of what Cloud Security Alliance teaches and so forth. 

    Rothman: Yeah, that’s exactly right. So it was interesting. So I guess it was probably eight and a half years ago we partnered up with the Cloud Security Alliance and helped them build their curriculum for their CCSK certification. And that got us to be much deeper than your typical analyst in terms of having to play around with cloud, having to understand automation, having to build a number of the kind of scripts and – I’ll call it the inherent messiness of building out a stack in the cloud and really making it operational, because we were eating our own dog food. Our website, a lot of the stuff that we had prototyped was really demonstrative of the challenges that organizations face as they move a bunch of their stuff into the cloud, and really as they embrace a DevOps mindset of trying to be agile, trying to be innovative, trying to move towards more of a continuous type of deployment model. We really came across a number of the challenges these organizations have in terms of just really maintaining control, right? Once you have multiple DevOps teams doing things in multiple kind of accounts, you start to run into challenges relative to how you enforce best practices.

    And that’s really one of the challenges that security people have with this whole DevOps revolution, which is, “Hey, this sounds great, but we’ve got a playbook, a set of practices and policies that we really require all of our environments to adhere to in order to protect that data.” And obviously the faster you move, the more agile and continuous you try to operate, the harder it is to enforce those best practices.

    Shimel: More about that in a second. You touched on two things there that we’re going to dig into. Let me just finish laying my groundwork. Mike, besides you, Rich and Adrian, the Securosis gang, there are at least two other key people on the founding team of disrupt:Ops who also happen to be friends of ours for years –

    Rothman: Yeah, that’s right.

    Shimel: Give them their due. Why don’t you give them a shout out?

    Rothman: You bet. So back when we spun the technology out of Securosis, we brought on two guys, very experienced guys in the security space. Our CEO is Jody Brazil. So Jody was a cofounder and long-time CEO of a company called FireMon, which I know you’ve done work with for a long time –

    Shimel: Sure.

    Rothman: So Jody kind of is really the driving force behind a lot of what we’re doing at disrupt:Ops. And our CTO is a guy named Brandy Peterson, another longtime security professional. He was actually the CTO of FishNet Security for 15 or 16 years.

    Shimel: As was Jody before Brandy, right?

    Rothman: Yeah. I met both of them around the same time.

    Shimel: Yes, when I first met Jody, he was CTO and FishNet, and he said, “Who is this Jewish guy with the New York accent?” I went out to visit him in Kansas City to tell him why you shouldn’t use Qualys, and he should use StillSecure, and he made fun of my accent. [Laughs]. Good times. Anyway. But Mike, let’s turn back to a couple of things you were saying about kind of the genesis driving disrupt:Ops. You touched on one thing; you said, “multiple DevOps teams.” So this is something we see, and I call it almost second-generation DevOps. You know, when a company or an organization is first embarking on their DevOps journey and they do sort of a trail balloon, and they pick a project and they have a DevOps team – you know, and they may not call it DevOps. They  have a team that’s practicing DevOps sort of principles on this one particular project. And oftentimes, wow, great success, because you are – you know, it’s an isolated environment, right? In some ways you’re not in the quote/unquote real world.

    And then you say, “This is great. Let’s take it out. Let’s expand that beachhead. Let’s expand the bubble.” And all of a sudden – you know, larger organizations; small teams or smaller organizations not so much, but larger organizations – now you start having two, three, four, six different projects that are being run with a DevOps mindset and you have multiple instances of Jenkins and you have three different – this team’s using Puppet; that one’s using Chef. And serverless. And all of the complexities that we see in today’s modern software factory. And it’s chaos, right? And then at some point the CIO or VP or director of IT says, “Oh my God, this DevOps thing isn’t working so good. It’s all over the place.” Let alone the security guy’s screaming, you know, right out of the Phoenix Project, “No, no, no!”

    Rothman: That’s right.

    Shimel: And that’s a problem that we see with a lot of second-gen, if you will, DevOps installs, where it goes beyond the initial team to several teams.

    Rothman: That’s right.

    Shimel: That’s what you guys see.

    Rothman: We do. And it’s interesting, because what you have are really kind of two paths at that point, in terms of how these teams start to behave when starting to be put under scrutiny in terms of, kind of again, their ability to enforce operational and security best practices. You know, one group kind of buries their head in the sand and continues to move forward and really don’t do much, right? They’re like, “Oh, it’s okay,” and you know, they’ll maybe run a vulnerability scan to kind of prove that they’re sort of within policy, but they really don’t have an operational environment that is built to work in concert with the rest of their factory. I love that term, right? That really does encompass a lot of how software is being built today. And then the other part ends up being folks starting to build their own scripts, right? They take maybe an open source-looking thing, they start to build a bunch of their own scripts and code around it, and then what they find is they’ve got a suitcase full of these scripts that they end up having to maintain, upgrade. They don’t have any means of reporting or any scheduling for how they enforce and run a lot of these scripts.

    It’s really – you know, we call it automation, but it’s manual automation, which I get is a total oxymoron. But that’s really how it is. You know, you have a bunch of folks, they’re building these scripts out, they’re executing them manually or through very simple triggers, without any means of maintaining, upgrading or scheduling or reporting on what these really key aspects of their operational environment are doing. And it becomes, again, a big problem, where you have – it’s unwieldy because of just the sheer number of things that are happening in a modern DevOps environment with multiple teams. Then you compound it with a number of very tactical scripts that are built to surround the environment, if you’re doing anything. And then at some point somebody says, “Hold on a sec, folks. This is just not where we want to be. We really need to think about it more from an operational excellence perspective and make sure we’ve got an environment that we can enforce our best practices for security and operations.”

    And that’s kind of when they get the sense that, “Well, I’m either going to have to make a significant investment in this and become like” – Rich calls them the unicorns of cloud land, right? You know, the Netflix and the Capital One and, you know, the folks who have whole teams of environments that are building tools to do this. Or they have to start looking for some kind of commercial option, because there really is no option, but you’ve got to do something. You can build it yourself or you can look to buy it. And candidly, until the last handful of months, there really wasn’t a realistic commercial offering to start implementing some of these capabilities.

    Shimel: No, no. I think the bigger – you know, a lot of what you’re saying certainly obviously rings true. The other fundamental piece of this is, even if you are using one of these homegrown, cockamamie, you know Rube Goldberg kind of things –

    Rothman: Yeah.

    Shimel: – is it the security guys who are? You know what? Shame on me. It’s not the guys. Is it the security team who’s doing this, people? Is it the DevOps cross-functional team? Is it a little bit of both?

    Rothman: Yeah, yeah.

    Shimel: You know, how many of – look, here’s my basic, fundamental belief. You are not going to find a developer who raises his hand who says, “I want to write shitty code. I can’t wait to put something insecure out there.” No one says that. No one says that, right? They’ll tell you they’ve got time pressure, there’s pressure to release, pressure to release, they wish they could do more on security. And some of them, the enlightened ones, as you said, they’ll even try. They’ll run a vulnerability scan, because there are certainly open source kind of tools out there for them to mess with.

    The problem is, they go to a lot of the security guys, and as you alluded to before, security guys say, “Well we have our rules and procedures here. This is a no.” Right? And they don’t want to hear no. They’ve got to go.

    Rothman: You know, what’s interesting, Alan – I want to hit on a couple things that you alluded to there, because they’re important, right? The first is kind of where you start to see the agitation for better operational and security practices. And it’s not that it’s not the security folks. Obviously they’re always going to push for that. But it really is – you mentioned it, right? The DevOps teams that we’ve worked with, they’re very cognizant, they’re very serious stewards of kind of the corporate data, and they want to do the right thing. But in a lot of cases they don’t know what to do, right? So you have a whole bunch of folks who don’t know what they don’t know and they don’t want to go to the internal security team because of a lot of those things you were just alluding to. They’d say, “No, we can’t do that. We have all of these rules.” And in a lot of cases the security team has not updated their set of conditions, their set of policies – 

    Shimel: Exactly.

    Rothman: – to work in a DevOps type of environment, so these folks really struggle. They talk to – again, on the consulting side of our house at disrupt:Ops, we do a lot of work with those teams in terms of helping them understand what cloud native architecture looks like and how to integrate cloud security into their environment, and really just to help them build a program in order to do that. So then you do that with one DevOps team, then you end up talking to another DevOps team and maybe the third DevOps team, and finally IT, at some point, they’re like, “Um, what are you telling our folks? They seem to like what you’re doing and they seem to be doing things in a more secure way. So can you help us?” And then IT starts to get religion. The security team starts to get a little bit of religion about what they have to evolve to. And it really is kind of that interesting transition where it has started kind of out in the DevOps community with, again, those enlightened ones going, “I don’t know what I don’t know. Let me get some help,” and then it kind of back-ending into security. That’s not always the case, but probably three quarters of the time, that tends to be the transition that we see.

    And the other thing I want to hit is just about velocity, right? JA very forward-thinking DevOps leader that we’re working with, he said it the best. And he’s just like, “I’ve got 20 DevOps teams. I cannot slow these people down. They are doing stuff; they are making progress. The last thing I can do is put a bunch of onerous security requirements on what it is that they’re doing. They’ll ignore me, they’ll go around me and that’s counterproductive. I can’t do that. What I really need is to be able to build and enforce a set of guardrails around their environment.” Right? That term, “guardrails?” It just really kind of opened up and was a light bulb in our head when we were having this conversation. It’s not about stopping them from doing things. If anything, it’s about allowing them to go faster, yet making sure they don’t drive the car off the road, right?

    And that’s really the metaphor that resonates relative to what we’re doing at D:Ops, and really just kind of where the environment is going, in that we have to be able to move faster. Slowing things down is not an option, yet you want to make sure that you are not putting corporate data at risk. You are not opening up significant new attack services just – because people are moving so quickly. Sometimes they make mistakes. Sometimes automation isn’t done in the perfect way. Sometimes humans get involved and certainly are fallible from that standpoint. But with a lot of the inherent, underlying automation orchestration that’s possible within the cloud and DevOps and continuous, you have options to enforce these best practices in a way that we couldn’t do in our traditional data center with our traditional waterfall types of development and operational motions.

    Shimel: Absolutely. And you know, Mike – I mean, I’ve spoken about this a lot at various conferences and stuff. A lot of that has to do – you know, we use this term “shift left” in security and all that. A lot of it does have to do with getting to security earlier, faster. Yeah, we’re going to have to automate it. Yes, we can’t be the roadblock. But we can be the appendix after – when I say, “we,” I mean security. We can be the appendix after the delivery or after the deployment.

    Rothman: I’ll raise you. I see your point and I’ll raise you one, right? I’m of the opinion that security isn’t a thing. Right? Security should be within everything.

    Shimel: Yeah.

    Rothman: And that’s really the mentality that we’ve been trying to push on the industry for a long time.

    Shimel: Amen.

    Rothman: And it’s one of these things where we really can’t afford to wait and be in a position where we’re trying to retrofit poorly architected systems that are moving faster than we’ve ever seen before and then say, “Oh hey, we’ve got to change it because of security,” or because of any other operational gap that we see there. Right? What we have to do is be building in a way that allows us to use architecture as security control, right? Isolation, segregation, multi-factor authentical, identity management – all of these things that are fundamental building-blocks of what we do in the cloud and DevOps can be utilized to build in a much more secure fashion. And it’s really about those kinds of practices that are going to make the biggest difference relative to what it is that we see and what it is that we deploy over the next couple of years.

    Again, like I said, I don’t think security is a thing five to seven years from now. Because if it’s not embedded within everything that we deploy, we screwed up.

    Shimel: Yep. You know, I heard – because I never come up with original ideas. I just listen. But I heard something awhile back – “Security needs to become synonymous with quality.” And we’ve had QA as a concept in software technology forever or for a long time. Security needs to be just built in, like as part of the quality control. Right? The same way shit or stuff gets tested, whether it be hardware or software or whatever. Security needs to be part of that quality. 

    Rothman: You bet. To use a software term? Right? Why is a functionality defect different than a security defect?

    Shimel: Exactly.

    Rothman: Right? And they should all be within the same thing. And again, that’s a lot of when you start talking about immutable, when you start talking about kind of building security testing into your pipeline, which again, is stuff I know that you guys at devopsy.kinsta.cloud write about quite a bit. Again, it’s really not any different than you managing and making sure that you hit your functionality kind of specs from the standpoint of just building code, right? Part of that factory. So yeah, the code has to work, but it also can and should be secure, and there really aren’t different operational motions to make that happen. And that’s what’s so exciting. For a guy who spent 25 years trying to retrofit and get people to think about security as part of their core structure and to treat it as really a business function, it’s really enlightening to be in a situation where you actually can implement a lot of those functions within the core processes that you use to develop the technology.

    And again, are we there yet? Of course not. Is there a lot of education to be done? Without a doubt. But we at least have the option now, right? That’s the aspirational goal to what it is that we all do every day, which is to make sure that we’re pushing for the right types of decisions to be made as early in the process as we can, because we all know how much it costs to fix it later.

    Shimel: Absolutely. Mike, I’ve got to tell you, listening to this conversation, I’m thrown back to, “NAC: is it a feature or stand-alone?”

    Rothman: You’re dating both of us, man. Stop that out. Cut that out.

    Shimel: It’s such a pleasure to talk a little theory here with you. But let’s turn for a second to real stuff. Not that what we’re talking about isn’t real. But I wanted to give our audience a little bit of a taste – so disrupt:Ops – great people, great big problem to tackle out there. Give us some specifics. What are you guys doing?

    Rothman: Yeah, so what we’re doing is we’re basically building a cloud management platform to implement a lot of these ideas that we’ve been talking about for the last couple of minutes, Alan. Initially it’s a guardrails platform. So what we have is the ability to, in an AWS environment, enforce a set of best practices across security, operational and also cost management. So one of those things we don’t talk about too much is that developers don’t really like to shut their stuff down because, “No, it’s going to mess with something. I don’t have to. The CFO bitches but we don’t really care.” What we can do is also implement best practices in order for folks to clean up their room, right? We try to do that with our kids all of the time, too. You know, so we try to do that with our developers as well.

    So really implementing a platform for security, operational and cost management guardrails, to again, immediately not just find the issues – there are a lot of tools, as you said. A lot of vulnerability scanners can kind of look at your cloud environment and tell you you have problems. The issue with that is that most operations people are overwhelmed with the number of other things they have to fix every day. Adding more stuff to their list doesn’t really help them. So disrupt:Ops is really focused not just on finding the issues, but also fixing them and enforcing those best practices. Let me give you an example of that. So let’s just say some person decides they want to open up a server to the internet because it’s just easier for them to do that. Well obviously that would be a pretty significant security problem. So as opposed to just taking that server offline, what you can do is you can basically change the security group to make sure that it’s only accessible via a known, good IP address range within your corporation. That’s a good example of a guardrail. Or making sure that you’ve got MFA turned on for every account that gets spun up by your magical Terraform environment that may not build that out as a core security requirement.

    So again, there are 50 to 60 of these guardrails, in effect, that you can implement in your environment, to again, make sure that you are not just alerting when you get out of policy, but actually fixing the issues, in a lot of cases before you even know they’re issues, so that you’re not sitting there trying to pick up the pieces after half of your data is in Chechnya.

    Shimel: Exactly. Or Romania.

    Rothman: Or Romania. It’s somewhere.

    Shimel: Mike, how is the offering packaged? Is it a SaaS kind of offering? 

    Rothman: Yeah, it is. What we found is since – you know, remember the days where people were like, “Oh, I don’t know about this cloud thing or SaaS. I’m not sure I’m comfortable with that”? Now it’s like, “Wait, you want me to install software in my environment? What? What’s wrong with you?”

    Shimel: That would be kind of oxymoronic for a DevOps tool, but –

    Rothman: That’s right. That’s right. So we deliver as a SaaS service. It’s actually a very cool kind of one-click provisioning process where, when you’re logged into your AWS account, you run basically a cloud formation script that really kind of gives us cross-account privileges to your environment. And again, we’re security guys, right? So at the end of the day, we are basically not even just assuming but enforcing least privilege. So we only have read-only access. When we need to make changes, we grant the privilege as we need it, then we revoke it. Because again, what we’re trying to do is make sure we enforce operational best practices; not add more attack service. So again, we’ve built the environment to be secure by design. We use all of our own dog food, to make sure that we are adhering to kind of common best practices. And again, we’re all 25-plus years security folks, so we do take that pretty seriously.

    Shimel: Excellent. Mike, for people who maybe want to download – not download, excuse me – sign up for it or check it out more, where can they get more information?

    Rothman: Yeah, disruptops.com. So by the time you hear this little interview, the site will be live. There’s a place you can sign up. You know, first we start with a demo and show you what it’s about and then you can sign up for early access. And again, we’re kind of expanding towards the end of this month, in October. We’re going to add a bunch more folks to the system with the goal of kind of being ready to actually sell the product by the end of the year.

    Shimel: Fantastic, man. Well, I don’t have to tell you, Mike. I wish you, Rich, Adrian, Jody, Brandy and the rest of the team nothing but success. We’ll be checking in, I have a feeling, pretty often here, whether it be through Security Boulevard, devopsy.kinsta.cloud, Container Journal or what have you. And you know, fantastic. It’s good to have you back in the game.

    Rothman: Yeah, no, I’m excited. Again, really it’s not just a big promise and not just a huge opportunity. You know, and what Rich, Adrian and I, and now Jody and Brandy have always done is – you know, we’re trying to do the right thing to evolve the practice. Right? And that’s what it’s about. It’s a big world and there’s a lot of opportunity. There’ll be a lot of folks talking about guardrails over the next year. And that’s great, right? Because what we want to do is get to a position where we can move faster but move more securely, and I a lot of cases, in the world that we grew up in, those were counter indicators. Right? You couldn’t move fast and be secure. You couldn’t do operational excellence and make sure that you’ve got a lot of these things enforced. And now you can, and that’s what’s really exciting to us. Again, we’re just excited to partner with you, as always, and again, just go out and solve some bring problems for a lot of organizations.

    Shimel: Fantastic. All right. Mike Rothman, president and co-founder of disrupt:Ops, a hot new entrant in the DevSecOps space, with a great pedigree and a real reason for being our guest today on DevOps Chat. I hope you’ve enjoyed this. This is Alan Shimel for devopsy.kinsta.cloud. Until next time, everyone, have a great day. Bye-bye.

    — Alan Shimel

  • Developer Trend Chat: Microservices, Node.js, Kubernetes

    Developer Trend Chat: Microservices, Node.js, Kubernetes

    The role of software developers has never been more important. Developers touch almost every corner of our lives, from health care and retail to education. It’s an extremely rewarding career—so much so that software development is now ranked the Best Job in the United States.

    The nature of software development demands that developers be fast-moving. New tools and technologies are providing developers the ability to adopt new skills, work faster and reshape how they work. In this Q&A, CEO of LunchBadger Al Tsang discusses trends in software development.

    Q: Data distribution is still a difficult and complex aspect of breaking down a monolithic architecture. How can enterprise architects avoid this or what is a good strategy for dealing with this snag when they will encounter it eventually?

    Al Tsang, CEO, LunchBadger

    Al: The data’s location, its dependencies and varying levels of data normalization, make breaking down monolithic architectures a challenge. Microservices initiatives can address these challenges by constructing a comprehensive data model. In the past, we had UML tools such as ERD diagrams. These tools are not as common anymore, however, the practice of what they represent still brings value in understanding what data your applications currently use, what’s obsolete, what’s redundant, and what’s related with dependencies.

    Microservices and the tie in to data should be atomic enough so that the data that it operates against is wholly owned and contained. This is easier said than done. There are often dividing lines that can be carved out that mirror the business functionality itself—whether that be by domain, by entity or by business process. In some cases, it’s a combination of all the above.

    Q: Why do you think serverless and FaaS (functions as a service) is growing at such a rapid rate and what specific pain points do serverless platforms (such as LunchBadger) address?

    Al: I believe serverless and FaaS is growing rapidly for a number of key reasons:

    (1) Infrastructure and application platform [are] becoming highly commoditized where a developer can pass the buck to have operational concerns placed on a third party.

    (2) The pressure and need to innovate requires an intense focus on working on your core IP as a differentiator while everything else in your code is a supporting role and oftentimes something that is completely boilerplate (e.g. think persistence to a database or NoSQL data source).

    (3) The movement and drive behind attaining microservices means getting down to bite-sized pieces of functionality means whittling down your logic do distinct reusable compartmentalized pieces of code—i.e., functions—and “serverless” fits this paradigm nicely by adding that you should only concern yourself with the piece of code on an event driven basis.

    (4) New and old companies alike have amassed a zoo of technological languages and platforms where skill sets and resources can be more efficiently utilized if you have a common runtime paradigm and can write your business functionality that supports polyglot usage.

    Q: According to a survey in 2017, Node.js was primarily used as back-end infrastructure for APIs. One of the main strengths of Node.js is that you can use the same language on the entire stack. How do you think Express.js and other minimal frameworks like it contribute to its popularity and why?

    Al: One reason Node.js has had rapid and wide adoption is because it is a massive equalizer. Node.js developers don’t need to have ninja-level domain expertise in high scalability, multi-concurrent connectivity, threads or similar complex topics just to be able to build a scalable app.

    Node.js takes advantage of a language that had the principles of high scalability built inside, namely evented asynchronous programming. Node.js also has an advantage as it’s the language that any web developer would have exposure to thanks to the ubiquity of browsers. Moving the same language and its strengths to the server was a huge win for folks to truly become “full stack.”

    Express.js has further contributed to Node.js’ success by adding the basics of web utility to Node.js. Every web application needs a web/app server that exposes URLs as routes. All servers have consumers making requests and processing those requests can often time be chained from one piece of logic to another. Express took the learnings from Ruby and Sinatra and brought them to Node.js.  You scaffolded a template project and filled in the blanks, as needed and where needed. It’s highly productive because the starting point for many web applications follow this same model.

    Express Gateway galvanizes the success of Express.js—its rich ecosystem of middleware modules and ubiquitous understanding among many developers and make its cloud ready and cloud native by separating out the configuration, metadata and conditionality to be declarative and dynamic. By doing this, Express Gateway runs in containers and orchestrators in a distributed way without worrying about being coupled at any given tier of the application.

    Q: Node.js and Kubernetes is a popular combination, but many developers and companies are confused on where to start or how it will affect their tech stack. Can you explain the top roadblocks you’re seeing and how companies can overcome them?

    Al: Roadblock #1: Where should you extend Kubernetes or where should you extend your Node.js application?

    I think this entangled mess is actually not as bad as it’s often perceived it to be. If it’s a core part of your application and you’ve written it in Node.js, then it should be completely independent of where and how you run it. Before the wide adoption of containers, many Node.js applications were being run in process managers. Many are still running in process managers within a container. In short, if it’s a core piece of your application logic, it should be in Node.js. If it concerns how and where you run the application logic, now you’re looking at configuration or extensions within Kubernetes.

    Roadblock #2: Should I use Node.js to orchestrate my application or leave that to Kubernetes and why?

    Orchestration means wiring up your application at the infrastructure level. Leave this piece to Kubernetes, which can expose and manage your microservices truly as “services.” When you’re writing Node.js applications, you shouldn’t have to worry about the “I/O” of how different Node.js processes interoperate. The whole premise behind microservices is that there are distinct lines drawn where processes must interoperate without assuming the other process may even be up and running.

    Roadblock #3: What if I have a component that has both infrastructure AND application concerns?

    This is a tough one where a lot of companies, large and small, are feeling pain. Kubernetes has a notion of an ingress controller, which is really nothing more than routing and exposure of requests externally to a resource designated internally to handle it somewhere with the Kubernetes cluster. This raises the question: Should you put application-level concerns on that ingress controller?

    I believe in a clean separation of concerns, even if that means having some runtime overhead. If your business requirements don’t require super low latency or other cases that drive you to blur the lines between infrastructure and application, then it’s much easier to maintain if there is a clean separation. This is really no different than slicing and dicing your application into smaller microservices. It’s just executed in your stack at a horizontal rather than vertical level. Build an ingress controller that is only concerned about routing to the right resource. Then build a separate API gateway that sits in front of the microservice(s) to take care of application-level concerns.

    — Malia Powers

  • Software Development: A Better Way to Measure Success

    Software Development: A Better Way to Measure Success

    Here’s a metaphor for software development you probably haven’t heard before: It’s like flying a plane. You have a starting point and a destination in mind, there’s a good chance you’ll change course midflight, and … sometimes you get a little nauseated?

    Okay, it’s not the best analogy. But there is one aspect of piloting a plane that offers valuable insight into how to measure your team’s performance more consistently. It’s called the performance-control technique, and it may be the best method you’ve never heard of for keeping your engineering teams aligned behind a common goal.

    Imagine for a second sitting in the cockpit of a plane, surrounded by clouds. Forget using the ground to orient yourself; you have nothing but your instruments and wits to guide you. It’s a situation I’m plenty familiar with, having taken flying lessons out here in rainy Seattle. In those cases, you use your altitude and power settings to set up a particular control scenario that results in your desired output, whether it’s ascending, descending, straight and level or turning. And then you monitor your performance—in this case via altitude, vertical speed and airspeed indicators—to verify that you’re achieving your desired output.

    Now unless your software development pipeline is irredeemably broken, you are probably not flying completely blind as you work to build and ship new features. But the technique still applies: Choose your control metrics (time to release, for example), and then set up a series of performance metrics to keep it on track.

    Cloudy Skies Ahead

    The greatest strength of the performance-control technique is its acknowledgement that any control can result in unintended consequences. Take reducing the time from check-in to release. It’s a fantastic goal: The quicker you get code into production, the quicker you deliver updates to your customers. The quicker you do that, the quicker you get feedback and respond to their needs. The quicker you do that, the quicker you can iterate on features. And the cycle continues.

    But it’s also a goal that can result in unintended consequences: When you single it out for attention, it’s only natural team members will want to do everything they can—at the expense of almost everything else—to achieve it. That’s not to say they want to do wrong; rather, they’re just trying to be good citizens. (And, okay, they don’t want their name to show up in a report about why the team didn’t meet its quarterly goal.)

    So people make compromises. Maybe they drop full regression passes and replace them with incremental feature and integration testing. It’s not that they don’t care about quality. But in their quest to help the organization meet its single-minded goal of reducing the time from check-in to release, they opt for the easiest change that could get them there, indirectly prioritizing speed over quality in the process. And that happens because there aren’t checks in place to keep it from happening.

    Taking Control

    How do you put this into practice at your own organization? Start by choosing your goals and then selecting the control metrics. For many, it could be as simple as increasing speed; it’s a natural goal to chase, for the reasons already discussed. But it may be something different; only you and your teams can decide by being intentional in how you evaluate your production environment.

    Then it’s time to choose your performance metrics for each of your goals. You can do that yourself, but it may be more illuminating—and, honestly, more fun—to get your software development team involved. That could range from encouraging them to brainstorm the negative consequences of implementing that goal to straight-up asking them how they’d game the system.

    The following chart offers examples of control goals, their resultant unintended consequences, and the performance metrics necessary to keep them in check.

    Desired Output Control Goal Performance Goal Unintended Consequences
    Climb to 4500’ Increase power, nose up Airspeed, Altitude Stalling
    Reduce bounce rate Reduce site latency Revenue, Conversions, Customer Engagement Smaller pages, poor experience, feature attrition
    Generate more conversions Increase availability Releases, Backlog Ticket aging. Lower frequency of releases, cherry picked work items, no risk taking.
    Reduce MTTR Speed up ticket closure Re-open rate, new ticket rates. Premature closure, delay in opening tickets for issues.

    It’s important to note that how closely you monitor each of these performance metrics will vary, and in some cases significantly. Rather than attempt to apply a one-size-fits-all approach, instead think of each in terms of how long it takes to detect that the metric is falling out of its control envelope and how long it would take to recover if it did. If it takes one day to detect that you’re outside of your envelope and three weeks to recover, then by all means, check in daily. But if it takes three months to fall out of the control envelope and just a day to recover, you may only need to check it once every couple of weeks.

    A Soft Landing

    As we’ve supported software development teams in their efforts to modernize their software delivery methods, we’ve heard time and time again, “What’s the best metric for measuring success?” And we struggle to come up with an answer because there is no golden metric that works for everyone. But more important, even the good ones can lead well-meaning engineers and team leaders to optimize for one thing while letting others suffer.

    So rather than search for a magic bullet, the secret is to find goals that work for your needs—and then build in metrics that keep you and your software development team accountable.

    — Rob Duffy

  • As Business Speeds Up, Testers Must Adapt

    As Business Speeds Up, Testers Must Adapt

    It should be no surprise to anyone working in software today that delivery speed is the name of the game. Irrespective of business sectors or B2B/B2C focus, global competition in application development and delivery has flooded the market. Due to the increased number of options in the marketplace, users expect an excellent experience from your application or site—and if you cannot meet expectations, they’ll find it elsewhere.

    A company’s ability to be highly responsive to (if not proactive about) customer’s needs is also valuable for adoption and branding. Established companies understand that delivering high quality in their products and services is core to brand value.

    This shift isn’t always easy for testers. Moving to a faster pace of work can be unsettling when your job is all about assessing the perceived quality of the software. Testers like to be cautious and check everything twice. The current work environment has made testers more germane than ever in software and IT organizations.

    Testers help build confidence in the quality of an application, and this perception of quality is what typically drives user adoption. If two applications have more or less the same functionality but one is faster, more reliable and has a better UI and design, you can guess which one will be more popular.

    Setting the Foundation

    To thrive in this new world, testers must learn to prioritize and focus on the most important needs of the team’s delivery goals versus testing every last mouse click. Testers will need to quickly incorporate feedback from multiple places, and work well with stakeholders across the development organization and the business. Efficiency is imperative, and that means increasing the use of tools whenever possible.  The following framework to develop a world-class testing organization when your company is moving faster all the time can make the transition a little easier.

    1. Understand business goals. While this may sound obvious, testers haven’t always had to think about the bigger picture. With more competition from everywhere, everyone on the development team should have a solid understanding of customer needs, business goals such as to drive new revenue or grow customer loyalty, and how the competition stacks up for the solution in development.
    2. Define a testing strategy. This exercise begins with understanding the business goals as they relate to the product or project and, at a more granular level, the goals for each sprint. The testing strategy should include a description of the overall project, priorities for testing mapped out into phases, timelines, testing approach, tools and methods to use and how progress will be measured.
    3. Develop a risk management plan. What will happen if those business goals are not achieved? For instance, the business stakeholder might say the project must go live in four months with new functionality. But is this an arbitrary goal or one that is embedded in an actual customer commitment? What will be the financial impact if the deadline slips? Which enhancements are critical and which are nice to have from the user perspective? Once those particulars are documented (or discussed), the testing team can create the appropriate plan, prioritize and determine what resources are needed to execute it.
    4. Resources. Testing teams with mandates to move faster will need support in the form of new tools, training and even new hires. Software that testers (as well as others) use can include DevOps and CI/CD tools, ALM tools, reporting and analytics, UI and API testing tools, systems for recording and logging manual testing, tools for usability feedback and more. This can get overwhelming for startups and other companies that don’t have the budget to purchase new software. Fortunately, there are many credible open source tools that can fit the bill. This reality is also why development teams must understand well enough what customers really want so they can prioritize on which features must be perfect and which areas of the application could meet a lower bar of quality.

    A Word of Caution

    Testers are well-equipped to make all the changes required for the faster pace of modern software development. This shift has been underway for a few years now, and many testers are embracing the change and the elevation of their role on engineering and IT teams. However, as testing relies on more automated tools, there are a few things to consider:

    1. Avoid tools mania. Testers, like many technical people, love to try out the newest toys, but it’s important to justify a tool’s adoption before acquiring it. With too many tools, there’s just too much noise and complexity. That will counteract any speed and efficiency gains that automation supplies.
    2. Automation tools do not replace testing. It’s not necessary nor helpful to automate every testing task. Testers should view automation tools as enablers, not a replacement for the function. Automation is wonderful for reducing effort and giving testers time to work on value-added activities that can boost the overall user experience and usefulness of an application. Be sure to baseline test objectives before recommending an automation solution.
    3. Institute process around automation. Ensure you have defined a proper process in sharing artifacts that are produced by your automation tools. Artifacts such as scripts must be versioned with proper comments and stored in a central repository instead of on individuals’ machines. Defining and enforcing good development practices such as check-in/check-out process, scripting style guides and object organization with automation will ensure a higher return on effort in the long run.

    In a few years, this point may be moot. But there are still many testers coming into new roles from the waterfall mindset who may struggle in the transition.

    Testers once lived simply in the silo of sluggish procedures. They knew exactly when they would be called upon for action and how they would complete a task. Now, though, testers must put on their business assurance hats and work side by side with developers, operations managers and others early in the process. They will have to learn new ways to test, such as with behavior-driven development and test-driven development. They need to get more technical, given the push toward test automation.

    What hasn’t changed though is the tester’s quest for quality in software. Great testers will add the most value when they can be proactive about alerting others to brewing problems before they present themselves to the user.

    — Mush Honda

  • Why Offshoring Software Does Not Work in the Agile Age

    Why Offshoring Software Does Not Work in the Agile Age

    These days, even refrigerators can be programmed to tell you when you’re running low on a certain food or when something has expired. Washing machines, coffee makers and toasters all use software to bring a more connected experience to their users.

    And a lot of the technology being developed today is targeting large-scale problems, too. Artificial intelligence can help with health care, driving and learning; blockchain can help with human trafficking and tracking blood diamonds; and data analytics can shed insight anywhere from credit-worthiness to cardiac risk.

    As demand for such technologies has grown, however, the brainpower behind them has been spread thin. Canada, for example, is predicted to have 218,000 tech jobs open by 2020—but not enough people to fill them, and the United States is certainly not better off. This pushes companies to offshore to places such as India to meet their development needs. But while software offshoring can offer these companies much-needed affordable technical expertise, there is one glaring problem: Traditional software offshoring does not work great for Agile development.

    Here’s why, and what you can do instead.

    Challenges with Traditional Offshoring

    In the Age of Agility, the digital transformation of industries is generating unanticipated software opportunities that rely more on creativity and reaction time, and less on technical grunt work.

    Under Agile development, software developers must work in close collaboration with a cross-functional team to constantly learn from client feedback and deliver new code that can be tested with real clients every one or two weeks. This approach serves to speed the velocity at which a company can test hypotheses about what the client wants without wasted time and effort.

    During this process, iteration is key. Engineers will deliver a minimum viable product (MVP) and quickly test to see if it works. They’ll gain valuable feedback from real clients and gradually iron out the kinks until the product is market-ready. But with the traditional software offshoring model, a high level of flexibility and iteration velocity is very hard to achieve.

    Under this model, a company in Silicon Valley and its software outsourcing partner in Asia or Eastern Europe may have at most two or three hours of overlap to coordinate their work. Conversations between team members thus become limited, occurring either very early in the morning or late at night.

    In an effort to communicate more frequently, the U.S. team begins to generate documents with instructions and recommendations that the foreign development team can read when they begin their workday. However, this generates complexity, diminishes real-time conversations and the insights they produce, and makes iteration a paper-heavy process. This is not the fault of engineers in the United States or abroad; it is simply a natural restriction imposed by geography.

    But it can be avoided.

    Opportunities with Nearshoring

    U.S. companies at the forefront of the digital age are realizing that to make software outsourcing work, they have to ensure it doesn’t sacrifice their agility and capacity to iterate. For this reason, companies have begun engaging in nearshoring—that is, they’re outsourcing to places in which the time-zones more closely coincide.

    Under nearshore software outsourcing, engineers in the United States and their foreign development teams are both awake at the same time. This provides the opportunity for significantly more real-time communication rather than enduring the disjointed paper-heavy communication in offshoring. As a result, project ideation and the creative cycle of market feedback and iteration are potentialized, and the software engineering process can run in tandem between the on-site client and the nearshore partner.

    For companies in the United States and Canada, the benefits of nearshoring can be realized by looking to Latin America, which offers a rich supply of talent. In particular, Colombia, Mexico, Argentina and Brazil offer sophisticated services, cultural compatibility and strong communications infrastructures.

    Not only do most Latin American engineers understand English—the majority of software engineering texts and materials are written in English—but  they also often rank ahead of India in terms of quality. According to Stack Overflow, Peru, Colombia, Chile, Argentina, Mexico and Brazil each score higher than India in terms of “Average Reputation of Top Users.” And this is no surprise, considering that Latin America is home to various unicorns, including Argentina’s software development company Globant, as well as MercadoLibre and Despegar.

    Far-shore software outsourcing services will not go away, and will still be used for certain types of projects that involve application rewrites or application maintenance. However, when push comes to shove, the companies tracing the new frontiers of technology would be wise to look at Latin America for nearshoring opportunities. India may be marginally cheaper than Latin America, but the additional flexibility and speed-to-market gained by nearshoring versus offshoring has no price tag. In the age of agility, only leaders and effective innovators will thrive.

    — Alejandro Vasquez

  • What Andreessen’s ‘Software’ Essay Means to Development Today

    What Andreessen’s ‘Software’ Essay Means to Development Today

    It’s been nearly seven years since Marc Andreessen’s essay, “Why Software is Eating the World,” brought to light software’s role in determining the next wave of winners and losers.

    Today, his essay is still relevant. The past decade saw the consumerization of IT in business. It also witnessed the shift in software consumption from desktops to the web and to mobile apps today, reflecting a mobile revolution. Software continues to eat the world, having moved to a “mobile only” development model.

    This revolution is defined by disruptive companies including Airbnb, Uber and others that run with more leanness and agility compared to their 20th century counterparts. These companies have few physical assets, relying instead on software.

    Since Andreessen’s essay, the software development life cycle has expanded and evolved—from waterfall to agile processes, from testing against just a handful to thousands of mobile devices and from serving a regional to a global one user base, to name a few examples. Demands for speed and quality control have been key drivers, and as a result, the life cycle moves faster than ever. A record number of developers are writing a record number of programs. This expands each and every day as businesses small and large, in markets old and new join, the mad dash to create more software, better, faster and cheaper.

    Creating Software, Fixing Mistakes

    Indeed, the software development curve is evolving at breakneck pace. But while the cost of creation decreases, the cost to correct a mistake increases. How can developers and their organizations stay ahead of the curve? Here are three principles we’ve observed from studying more than 25,000 customers in the last seven years:

    Implement Modern Development Techniques Such As CI/CD

    Continuous integration (CI) and continuous delivery (CD) are two modern development practices designed to help teams release quality code faster and continuously. CI/CD has gained popularity over the past few years, and for good reason. When implemented effectively, CI/CD shortens delivery times, improves overall quality and enables teams to adapt quickly.

    Key to a successful CI/CD practice and ensuring quality software development is the practice of testing throughout the development process. The cost of fixing already deployed broken software is high, and it’s crucial to ensure your software works properly on all platforms and devices to minimize customer abandonment. This is especially true when you consider that the cost of regaining a customer is four times higher than winning them in the first place. Getting software right the first time through a culture of continuous testing is critical to success.

    What does ideal testing look like? It is automated, runs fast and includes good coverage. As the already broad ecosystem of operating systems, devices and platforms expands, this last point is critical. What good is your app if it doesn’t run (and, by extension, monetize) properly when it gains popularity in some part of the world with users on a different platform from the one you tested on? Developers need to ensure their software works on as many different combinations of browsers, devices and operating systems as possible, to help their business reach the maximum number of users.

    Introduce ‘Self-Driving Software’ Practices, Technology into Your Development Cycle

    When everyone is running so fast, it’s critical to ensure the software you’re writing actually works, won’t break your infrastructure and will actually be used by consumers. The “garbage in, garbage out” model is not an option, especially given the expense of creating software today. While the barriers to create software have lowered over time, the bar has risen in terms of creating something really remarkable.

    The good news is that we are at the cusp of attaining the holy grail of software development. I have interviewed multiple engineering and quality leaders at many of our customers who are pushing the limits on ultra-efficient DevOps. Through a streamlined process (which, frankly, is nothing short of beautiful), they are able to create a single, seamless flow between development to testing to operations to deployment and monitoring—without compromising on speed or quality. The key is to prioritize speed over cost and quality and agility over all else. If an organization isn’t following this principle, they risk being left behind. In the “Stack Overflow developer survey 2017,” more than 65 percent of developers indicated that they follow some kind of agile development methodology at work.

    The end result is deploying software almost immediately as soon as it’s written, for release as quickly as possible. Companies strive to achieve this scenario, and the highest performing organizations have already reached this development dream in some manner or another. Attaining CI/CD and automating all the tasks involved in creating a test system in real time is one of the arenas in which the next big software battle will be fought. The quest for the holy grail of software development is on.

    Support (and Attract) Your Engineering Talent by Providing Flexibility

    In  the mad dash to develop amazing software, engineering has become among the hardest roles for companies to recruit for.

    Organizations need to be nimble enough to attract top engineering talent. Part of being attractive to potential hires is understanding that software can no longer be monolithic. More than ever, the talent pool expects choice in the languages, platforms and tools they can use to develop and work most efficiently. In the Stack Overflow developer survey, participants valued the languages, frameworks and other technologies they’d be working with as a critical (4/5) decision factor while assessing potential jobs. This doesn’t mean providing a “free for all” for developers to use everything and anything they want—rather, it’s all about adopting flexibility. The companies that are most attractive to the best engineers follow this principle.

    Conclusion

    Today, we’ve moved from software eating the world to the world eating software. Developers simply have no control or knowledge over where their website or app is going to be downloaded or accessed from. Unless it’s an internal app behind a firewall, any software developed today by definition is consumed by the global community.

    Even if you test for all scenarios and devices, the future will always bring new devices, browser versions and operating systems. Your customers may use these instead of what you initially tested for. One option is to test for emerging devices in an in-house lab; however, that means committing to resources, time and qualified engineers to maintain it. In most cases, this approach is simply unsustainable.

    Companies must focus on ensuring their software works both with what’s out there today and with whatever browsers, devices and platforms will be out tomorrow. This is a major challenge, requiring considerable investment. Companies need to decide whether they are in the business of testing software themselves, or leverage solutions that are.

    Well before Andreessen’s seminal article, Geoffrey Moore wrote “Dealing with Darwin.” Here he he expounded upon the concept of core vs. context. Organizations will be well-served in deciding (quickly) whether building an in-house testing infrastructure is core or context to their business. The alternative is falling behind—or worse, being eaten up by the competition.

    — Ritesh Arora

  • Why Your Software Supply Chain Might Be Your Achilles Heel

    Why Your Software Supply Chain Might Be Your Achilles Heel

    Historically, cybercriminals are as lazy as they are innovative. They can come up with clever exploits and attack vectors, but they still generally focus on the low-hanging fruit in the most target-rich environments. Recently, attackers seem to have shifted focus away from directly targeting companies with strong security or a wealth of resources, instead going after weak links in the software supply chain.

    No matter how many resources or how much effort you dedicate to securing your networks and protecting your data, it can all be for nothing if you trust a third-party supplier or vendor that is vulnerable. It is also true of your software supply chain, because the process of developing and deploying software offers opportunities that attackers can exploit.

    The Case of Xcode Ghost

    iOS devices are considered to be exceptionally secure. iOS is not impervious to attack or exploit, but it is more secure than the competing Android platform. Apps for iOS are also considered to be more secure, because Apple’s App Store is a “walled garden” with a stringent review process for an app to be approved.

    A few years ago, however, it was discovered that more than 4,000 apps in the Apple App Store contained malicious code. Attackers had figured out how to leverage a weakness in the software supply chain to get past the gatekeepers of the walled garden.

    How did they do it? iOS apps have to be written in Xcode. The Xcode software is provided for free, but the Apple servers are often slow—especially when trying to download from China, so developers often just search the web to get Xcode from a third-party site. Cybercriminals developed a malicious version of Xcode, then gamed the systems to ensure their version of the software would show up at the top of online searches.

    Apps developed with the malicious Xcode software contained extra code that would phone home and provide the attacker a backdoor—and an opportunity to inject malicious code or execute commands within the compromised apps.

    Low-Hanging Fruit

    The Xcode Ghost incident is an excellent example of a software supply chain attack, and a perfect illustration of attackers going after the low-hanging fruit. iOS is relatively secure. The Apple App Store is protected and apps are reviewed to ensure they meet strict criteria. So, instead of trying to go after Apple or iOS, attackers figured out how to strike the platform used to develop the apps and sneak in through the back door.

    I spoke with Ryan Olson, Intelligence Director for Palo Alto Networks, about the rising threat of attacks on the software supply chain. He explained that attackers know that when they are going after a hard target that would be difficult to defeat directly, there is a better and easier way: Just figure out who they trust and go after the low-hanging fruit.

    This becomes a potentially larger issue in the context of DevOps and automation. The recent NotPetya ransomware was propagated through a malicious update. If your systems or software are configured to update and/or deploy automatically, a software supply chain attack can be in the wild and compromising your applications or data before you know it.

    There are tools available to scan images, applications or containers before deploying them, but they typically test for known vulnerabilities or overt crashes or conflicts. They are not designed to look for latent backdoors that might be buried in the code.

    Olson recommends that you start taking a closer look at the software vendors you rely on. It is important to understand how they are securing and protecting their code—because their weaknesses expose you to threats by virtue of the trust you give them and their apps.

    — Tony Bradley

  • DevSecOps: Deception in Depth

    DevSecOps: Deception in Depth

    Mantraps, tripwires and tarpits … sounds like the start of a solid spy-movie plot, doesn’t it? These are among the many concepts of physical security that are making the crossover to software security.

    You’ve likely heard that security is all about defense in depth—the idea of layering several defensive measures so that their combined effectiveness is stronger than if they were used individually.

    In the case of deception, we are taking tried-and-true warfare approaches and applying them to DevSecOps. According to researcher, deception specialist and popular speaker Herb Todd, there is no rule of law in warfare, and in environments where you put code on the web, you are giving the world access.

    Deception Approaches

    There are two primary approaches to deception in software security. The first is to get attackers to look in places where you have no vulnerabilities. Todd would call this, “Appear weak where you are strong.” This is what honeypots do: offering an attacker a weak-looking target that appears as if doors are unlocked and there’s an opportunity to gain a foothold. All the while, there is no real vulnerability to be found, and the defensive team can be alerted to an adversary on the network.

    The second approach is to isolate or slow down attackers as they progress through your networks and application. One way to do this is to add tripwires throughout. When parts of your web application starts being manipulated in ways that normal users wouldn’t do, you know you have an adversary testing your application looking for vulnerabilities. At this point, you can move them off to a segmented portion of your application or network through the use of a mantrap, or slow them down with a maze or tarpit.

    Provoke Action

    Deception measures hinge on being able to get attackers to take an action. This is the main goal of deception. If we can provoke an action from the adversary, then we can begin to understand them and to track them throughout our system. This creates a feedback loop in DevSecOps for developers, operations and security to determine how many adversaries are currently interacting with the system, what they are attempting to do and how to add new defenses in the future.

    None of the major data breaches you read about start with the organizations saying, “Our security team has been tracking these attackers for weeks.” In fact, it is the exact opposite: We usually learn that the attackers had been inside the system and exfiltrating data undetected for months on end. The organization didn’t know they were under any attack until after their database got dumped on the internet or leaked to the media.

    Start with Tripwires

    Deception sounds great, but how do you go about it? In Todd’s talk on the Modern Security Series, he gave several ideas on how to get started in the case of web application security development. Methods in DevSecOps include adding fake paths to robots.txt and fake hidden fields to your application. When someone hits your tripwires, you log it and fire an alert for the team to investigate. As you gain familiarity with that, moving into tarpits and mantraps begins to make more sense.

    One example that Herb Todd gave in his talk was to add a hidden field of “encrypted-auth” and set it equal to “YWRtaW49ZmFsc2U=”. An astute web developer will notice that this is not actually encrypted. This is a base64 encoding of “admin=false.” When an attacker sees this, they will immediately think, “What if I tried sending admin=true?”

    So the attacker takes the string “admin=true” and then uses base64 encoding to produce “YWRtaW49dHJ1ZQo=” and submits that. Now, if your application receives a value for the hidden field that isn’t the original value or matches “YWRtaW49dHJ1ZQo=”, then you have actionable intelligence to alert on.

    When creating a tripwire is this simple, why stop with just one? Adding small tripwires throughout an application through DevSecOps can be highly valuable to identify more advanced attackers.

    DevSecOps: Making a Practice of Deception

    While deception is beginning to gain favor among AppSec professionals and thought leaders, it’s not yet standard practice by any means. But this should soon change, as  the next year or two see the best security teams adopting deception strategies—and for good reason. With traditional methods such as web application firewalls (WAFs) breaking down in the face of high-volume traffic and more sophisticated threats, security needs to become more creative. Defense-in-depth concepts such as deception have been proven effective in the physical worlds of warfare and spycraft—and when incorporated into DevSecOps, they can be just as powerful for ensuring protection in the digital world as well.

    — James Wickett

  • ‘Software is Eating the World,’ and DevOps Needs a New Infrastructure to Keep Up

    ‘Software is Eating the World,’ and DevOps Needs a New Infrastructure to Keep Up

    Every Fortune 1000 company should expect to face Uber-like competition in the next decade. These digital disruptors are software companies; they are basing their business models on software platforms, and development of these software platforms is much more agile than in the past. To compete effectively, every company must essentially become a software company, and it must adopt agile software development practices to keep up with the demand for new applications and services.

    Software Development: The Engine of Digital Transformation

    Digital transformation will be the mantra for IT initiatives in 2018, and the underlying technology will be agile software development. IT organizations are at a crossroads: They can either work to become more agile and innovative, or business unit managers will decide to expand their use of public cloud providers, reducing the role and value of internal IT organizations.

    Applications are defining and continuously redefining products, services, customer experience, business processes and competitive advantage. About the only thing not being redefined by applications is the IT infrastructure. As applications and organizations become more dynamic, IT largely operates at the pace it always has. High-velocity DevOps life cycles collide with an IT infrastructure that is optimized for stability and is too complex to keep up. The pace of the organization is slowed down by its applications, while the pace of applications is slowed down by their infrastructure.

    How DevOps’ Role is Changing

    For decades, DevOps organizations have been at the mercy of IT departments, having to request resources for development efforts. DevOps groups have endured long provisioning cycles, waiting days or weeks for IT to provision servers, databases, development tools and other resources. Even the tools for requesting resources are antiquated—most IT departments rely on manual ticketing systems that require requests to be passed from one silo to another before a request is completed.

    But with software development as a corporate priority, companies are now looking for much more streamlined development processes. DevOps now wants to drive the provisioning process. This has given rise to a lot of ad hoc use of public cloud resources by DevOps teams. No longer content to wait for the IT department to provision resources, DevOps organizations are taking control away from the IT department and are provisioning resources themselves with public cloud services such as AWS, Azure or Google. These platforms have enabled agile software development, and the spread of agile software development is integrating development and deployment into a unified continuous integration/continuous development/continuous deployment (CI/CD/CD) process. In the meantime, the IT department wants to maintain control over the infrastructure and software tools in use, as well as the monthly spend on public cloud resources.

    Unfortunately, what began as a fast, ad hoc way of acquiring compute resources from the public cloud has become a significant drain on enterprise resources. Monthly cloud costs are spiraling, and the IT department has little control over these expenditures. In many ways, DevOps has usurped IT’s role in resource provisioning. Also, it’s easy to create a sprawling, undisciplined IT environment where the company is not sure which workloads its developers have on AWS or other public clouds. Finally, the company may be developing applications that use intellectual property or confidential client information—or require high security or government compliance—so using public clouds for DevOps may not be an option.

    What Developers Want

    Clearly, today’s DevOps teams want to be able to select and provision resources as needed. They want to be free to build their own workspaces without waiting for provisioning by the IT department. If cloud platforms accelerate the application life cycle by applying new technology to operations and integrating and aligning applications and infrastructure, the challenge for IT is to regain control over the process. The answer is to deploy a private cloud platform that is as easy to operate and consume as a public cloud. With a self-provisioning private cloud platform, DevOps teams can continue to set up their own workspaces and access the tools they need to complete their development cycle, and IT’s private cloud—rather than a public cloud provider—makes those resources available.

    Compared to public cloud, private cloud delivers improved security and governance, higher performance and lower costs. It also unifies the separate compute, storage and networking silos to deliver a cost-optimized IT infrastructure. With a private cloud, DevOps can have the agility it needs while IT retains the control it needs. It’s a win-win for both groups.

    — Ajay Gulati