devopsy.kinsta.cloud

Category: DevSecOps

Dismantling Inefficiency: The DevOps Point of View

I recently caught up with Paula Thrasher, application delivery lead at CSC and DevOps pioneer in government circles. In our conversation, we discussed a number of important topics including cross-functional teams, software supply chains, paying down technical debt and her search for passionate people to hire. To succeed in managing large projects, she’s taking a different approach—one you might want to apply in your own organization.

Thrasher: I’m Paula Thrasher with CSRA and I work for a division company we call Delivery Services Organization.

Derek Weeks: What was your presentation at the Rugged DevOps conference about today?

Thrasher: We had a panel about the people, process and cultural aspect of DevOps, and the security context, as well. When you’re moving to the Ops transformations, what kinds of behaviors are both challenges and opportunities.

There was a comment at the beginning session about how any sort of transformation that relies on human nature has failed. So you can’t change how we work, but you can leverage how people are motivated to try and encourage the right behavior. I think that’s the thing that’ll resonate with the security professionals.

Link to Panel: https://www.youtube.com/watch?v=IbrY8mm9aI4&index=7&list=PLotLY1RC8HovW-XnZbItSBupjbTzFkfJw

Weeks: You’re setting up a new organization and have a lot of projects that you’re going to take on. Is it possible to achieve a lot with smaller teams? Do you need a big team to do rugged DevOps?

Thrasher: I think the main thing that we’ve been focused on is creating the team that is sort of the center of the universe. I think the most important thing is to have a team that is cross-functional. A lot of times when you think of teams … this is the development team, this is the Ops team, this is the security team … the question is, do you have all the players on the same team, or are they on different teams? I think if you find yourself in a situation where any one team has to deliver to another team, instead of actually being responsible for delivery of the actual business submission function, you’ve got some horrible web of dependency you’re going to have to untangle.

The big focus of what we’re trying to do in the new organization is build that completely cross-functional team and to enable them as a team to be placed on different projects, different missions. So instead of doing single-band rotations, where a single employee moves onto a new project, we’re actually trying to move full teams. As a team, they learn how they work best and how they can deliver. It makes a really powerful solution for actually solving problems. I’m really excited about that.

Weeks: A lot of people at the Rugged DevOps Conference are interested in the rugged DevOps phase because they’re concerned about being left behind by DevOps. Is that a real concern for security teams: getting left behind?

Thrasher: Yes, we definitely see a dichotomy in the world of how security interacts with development teams—between the traditional showing up at the end and inspecting versus really embedded security. It’s so clearly going to this idea that you can’t just inspect security. Security can’t be this checkbox at the end. You really have to build it in from a computer. You have to actually make design decisions about security. You have to make sure the developers know secure coding practices. You have to have tools that scan, check all the time, and get feedback. You can’t just add that at the end.

I think it’s a real challenge to security folks, just as it was for everyone—developers and Ops people—to change how you think to that systemwide view. You can’t just be a phase at the end. We actually have to be part of a whole equation, that whole systems thinking. I think that is so instrumental to what DevOps really is as a group.

Weeks: One of the things that you recently talked to Mark Miller about was an idea of using pictures to help with collaboration and to get everyone on the same page. Can you tell me about that practice?

Thrasher: Back to this human nature piece, I think humans are inherently really visual thinkers. I also think you’re asking people to bring cross-costuming across social teams. Well, if you’re a security person and you’re talking to an Ops person, you probably don’t speak the same language. Or if you’re a developer and you talk to a security person, again, you don’t necessarily speak the same language. But when you speak visually, you can understand each other better. You don’t bring your acronyms to the table. We’re talking about a diagram.

I also think it takes the personal element out of it a little bit. We’re talking about the system. We’re not talking about each other. The security team is not accusing the developers of being a bunch of renegades. They’re talking to them about a process and how they process it. It keeps the conversation where it needs to be, which is on making the process better and delivering better software.

Weeks: How is the growing interest in software supply chains changing the way you’re thinking about DevOps?

Thrasher: I think simplicity is an underused tool in security. Recently, one of our DevOps teams that was going in their rapid fashion had a deployment that had a vulnerability in it. They had to respond quickly, and they actually did respond quickly. But when we unpacked it, in retrospect, we realized: “How did we get here? How did we let this vulnerability get in?” We had all sorts of scanning. We had, supposedly, all the right things to have prevented that security flaw. Yet, how we got there was this Venn diagram of all these things overlapping. It was one of those security bugs that slips through the cracks.

We realized what happened is we had gone to over-engineering; we had over-engineered for performance. In doing that, we had overlooked the interaction of three components in a very particular circumstance. It was like a “every third Tuesday if you spun around three times” kind of bug. But I think that’s exactly what happens a lot. That’s super common in terms of those last defects, and there’s quite a bit that comes out of that overlap of complexity.

Fewer, more secured suppliers reduces the lift of having to upgrade updated components. It also means if you find an issue, your unpacking of that issue is a lot easier because you have a lot less things going on. All engineers need the reminder of “keep it simple.” But I think that’s a real powerful thing. Fewer, better suppliers, from a supply chain standpoint, is one of the mantras that I really appreciate because I just see it match. It really makes a difference.

Weeks: You’re working with a lot of government agencies across the board. What’s their interest in rugged DevOps practices, or just DevOps practices that you’re bringing into play?

Thrasher: I think in the government space, and it’s true for any enterprise, everything has to have security. There is not an agency that we can work with that doesn’t have privacy data, or heavily secured data, in the defense and intel agencies, or just straight-up vulnerabilities from being a target. The scale of the enterprise means there’s a lot of potential systems and a potential surface area for attack. You can think to all the high-profile attacks that have occurred in the federal government recently, and it’s made people really alarmed!

One of the things that’s very interesting is that the whole federal government did a “cyber sprint” last summer, where they did nothing but pay down technical debt, and get rid of systems and their vulnerabilities. It was wonderful! And I think every organization should do just that: spend one month a year doing nothing but eliminating old frameworks and applications. That really simple, easy thing often gets overlooked because businesses are always saying, “More, more, more, more, more.” However, we actually measured the metrics of the teams after the “cyber sprint,” and all of our teams’ velocity increased. After we let teams upgrade legacy components, all of them had an improvement, which is sort of like a “Well, duh!” But I think it’s a win-win really, that if we can allow ourselves to invest in paying down our technical debt we can get better results.

I think that in a government space, it just has to be part of the equation. So much of what we do is so instrumental to everything.

Link to interview: https://www.youtube.com/watch?v=5veGeDAaJSI

Weeks: As we started off the conversation, you have a new group. You have a lot of projects that are going to get underway. Are you hiring?

Thrasher: Absolutely.

Weeks: Who are the type of people that you’re looking for, if someone wants to get in touch with your company?

Thrasher: We are at CSRA.com, and if you go to our jobs and our careers website, you can see some of the openings we have. Actually, I’m really excited. One of the projects we did internally was building a whole new hiring model. We used big data. We built in some DevOps, and we’re taking advantage of our own technology to do that.

Like I said, what we are really trying to do is build good strong teams, and so that’s something we’re very focused on. So one of the pilots we’ll probably be doing shortly is a team-based hiring model where a team actually hires for themselves.

We’re looking for folks who are just really passionate about solving digital government. And I think that’s the main focus of us is really just being passionate about the mission we’re trying to do. We’re trying to help the government be that digital agency. Looking at the interactions we have with some of these Silicon Valley apps, how come we don’t have that interaction with the government? That’s what we want to create. So we’re looking for people who are passionate about our mission and willing and interested in solving problems. Hopefully we can build a great team.

Weeks: Excellent. Thank you.

If you loved this interview and are looking for more great stuff on Rugged DevOps, I invite you to download this awesome research paper from Amy DeMartine at Forrester, “The Seven Habits of Rugged DevOps.”

As Amy notes, “DevOps practices can only increase speed and quality up to a point without security and risk (S&R) pros’ expertise. Old application security practices hinder speedy releases, and security vulnerabilities represent defects that can leave a company open to cyberattacks. But DevOps practitioners can leap forward with both increased speed and quality by including S&R pros in DevOps feedback loops and including security practices in the automated life cycle. These new practices are called Rugged DevOps.”

August 30, 2016
3 Simple Steps to Rugged DevOps 101

As the CEO of WhiteSource, I regularly consult enterprises about building and enforcing their open-source security policies.

In the course of my consultations, I discuss open-source security protocol with many software companies and enterprises, and I’m often baffled by the fact that many organization aren’t boosting their software development performance and speed by bringing their development and operations teams together via DevOps. In fact, only 25 percent of enterprises are currently doing DevOps. I mean, if you were a CTO and someone said you could make your software development 200 times faster and reduce unplanned work/rework by 22 percent, wouldn’t you jump at the chance?

Now, I know DevOps still is not being implemented even though it has proven to improve deployment rates, so I shouldn’t be surprised that security isn’t joining the DevOps party. But I am. Just imagine the benefits of giving security the DevOps treatment. There’d be no more (or, at least, significantly fewer) last-minute security issues, no more unrealistic security policies, and engineers, IT and security professionals who actually understand each other and improve their own processes to work better together. This is the essence of rugged DevOps.

Implementing rugged DevOps is a huge decision for any enterprise, and it requires a lot of planning and time to implement. However, you don’t need to completely remodel your organization to start enjoying some of what rugged DevOps has to offer.

So, what can you start doing today?

Shift Left Your Security

DevOps is all about shifting left security to find security issues as early in the software development life cycle (SDLC) as possible. The earlier you find an issue, the cheaper and easier it is to fix.

So, how can you shift left your security? Start by integrating security tools into your continuous integration (CI) process to find vulnerabilities as soon as they’re added to your build and sometimes even your repositories. Tools exist that can integrate with your build tools and identify potential issues in your build.

Also available is a browser plug-in that allows developers to detect security vulnerabilities while they are searching for open-source components online. This tool helps developers say no to vulnerable components long before downloading or integrating them with their code.

Shifting left security is the hardest part of any enterprise’s rugged DevOps journey. Yet by taking some small steps, you can start to enjoy the benefits of DevOps with a twist of security.

Adopting a Micro Approach to Security

Just as operations has adopted a DevOps mindset to understand and become involved in development, it’s now security’s turn.

Security needs to stop being on the outside of development, looking to control it through detailed security policies. Instead, it needs to take an active part in it, steering it toward security. Security can do this by focusing on understanding development and deployment processes, and searching for possible quick wins.

From my own experience, every time a security team takes the time to understand the peer code review process, small changes are made with significant contribution to code security.

I also often notice that security teams are unaware that some developers may still use the old “copy and paste” method in using open-source components. This makes it impossible to track known open-source vulnerabilities. Once security teams understand this, they can improve guidelines to ensure they are not blindsided.

The good news is you don’t have to overhaul your whole organization to give security a micro-makeover. Your security team simply needs to start getting into the finer details rather than planning long-term security programs and road maps.

Speaking the Same Language

A good sign that an enterprise is doing DevOps is when developers and operations call software issues by the same name. This means the two teams have defined their issues and objectives together, and are working toward joint goals. The same is 100 percent true for rugged DevOps.

If development sees a problem in terms of unplanned work, operations sees a glitch and security sees a vulnerability, collaboration will be very difficult. Introducing a shared vocabulary for issues throughout software development is an effective way to get your professionals on the same page.

With this shared vocabulary, development and security can sit together and define shared key performance indicators (KPIs). For example, development’s goal of faster deployments can be linked with security’s aim of reducing mean time to release (MTTR). Once each team knows what the other wants to achieve, they can understand how to help each other.

At the end of the day, security is more likely to be taken seriously if it is understood. And this is key. With the typical enterprise ratio of 100:10:1 development, operations and security professionals, security needs all the friends it can get.

Ready to Try Rugged DevOps on for Size?

As you can see, you don’t need to turn your enterprise upside down or even have a rugged DevOps team to start getting some of its benefits. Just follow these three steps, and your enterprise can get a taste of what rugged DevOps has to offer.

So what about you? Are you ready to take rugged DevOps for a test drive?

About the Author / Rami Sass

Rami Sass is an experienced entrepreneur with deep background in R&D and product management. At Eurekify. and later CA, became an expert in designing and implementing complex security management and compliance software systems, and in delivering them to the market. Currently he is co-founder and CEO of WhiteSource.

August 18, 2016
DDoS Defense: Can You Tell Friend from Foe?
In many organizations, networks are at the core of the business, enabling not only internal functions such as human resources, supply chain and finance, but also the services and transactions the business depends on for revenue. That makes network availability critical. Any interruption of access from the outside world turns off the revenue spigot, impacting profit and creating a poor user experience that can damage customer satisfaction and result in customer loss. The worse the outage, the worse the damage. That’s why speed is so important in detecting, diagnosing and responding to denial of service (DoS) and distributed denial of service (DDoS) attacks.

One of the chief challenges in responding to an attack is to distinguish friend from foe. Without a way to drill down into traffic details and examine host-level traffic behavior, it can be difficult to tell the difference. Traditional network analysis technologies based on pre-cloud architectures have been too limited in their compute and storage capacity to do more than perform predefined alerting and summary reports. That’s just not enough information to really get to the heart of what’s happening in a complex networking scenario.

Fortunately, new big data techniques allow us to dig deep into huge volumes of network traffic details so that it’s possible to understand what is really going on. With a properly implemented big data platform, you can pivot your views of data to gain insight rapidly, in operational time frames, so you can act to mitigate an attack or remediate a more innocent but still painful network issue. We’ll examine data that is readily available through common network traffic flow telemetry exports such as those provided by routers and switches enabled by NetFlow, sFlow or IPFIX.

Starting at the Top

Let’s say we’re seeing symptoms of an attack in our infrastructure. We’ll use traffic flow summary data to quickly scan total traffic in bits per second just to see if anything stands out.

There’s no obvious traffic spike from this view, but then again the network we’re looking at is running at an average of 60 Gbps, so that doesn’t mean there aren’t worrying things going on at deeper levels of the network.

Analyzing Source Geography

One of the things that big data is good at is fusing many data sources together. By combining NetFlow data with GeoIP, we can look at traffic by source geography. In this case, the network doesn’t get a lot of traffic from China, so what happens when we filter total traffic by China as source?

A ha! That analytical pivot produces a graph above, showing two obvious spikes that are well above average. Below, we zoom in on the time of the spikes.

Analyzing Unique Source IPs

The spikes themselves are suspicious, but is this just a large data file transfer? We can find out by looking at how many different source IPs are sending traffic. To look at host-level details, note that we’ve gone beyond the point where you can use summary information. At this point, we are analyzing raw NetFlow record details.

Those raw NetFlow details sure are useful, because there is, in fact, a huge increase in the number of unique source IP addresses sending traffic to particular destination IPs. This tells us that we’re not looking at a large file transfer from a single machine, but a highly distributed set of senders. Botnet much?

Who’s Getting Hammered?

The next step is to to determine which IP or IPs are getting all this (probably unwanted) traffic from 14,000 or so individual host IPs.

The ability to dig into high volumes of host-level NetFlow details again proves its utility. We can see that the main target is a solitary destination IP address. There’s really only one likely explanation for traffic from thousands of hosts in a country which you have no business dealings with to a single IP that suddenly spikes from nearly nothing to more than 1 Gbps: This is a DDoS attack. Note that this isn’t a mega attack, but it still can cause real problems for whatever is running on that individual host and anything else that depends on it. If it’s your DNS server, the attack might make it impossible for lots of other servers and applications to function.

Going Deeper

Now that we know it’s a DDoS attack, we shouldn’t stop because we don’t know whether that attack is coming from other countries besides China. We pivot our analysis again to widen our lens.

Lo and behold, there is indeed DDoS traffic coming from multiple countries, including the United States, Japan, Russia, Sweden, China, Taiwan, Brazil and Estonia. Good to know.

Characterizing the Attack

Next, we want to know specifically what type of traffic we’re seeing and what that tells us. We’ll group the traffic coming from all those source host IPs by protocol.

Clearly, this is UDP-based attack. Now we’ll look at the destination port(s).

We can see that the UDP traffic is being sent to multiple ports, and it’s obvious that we’re experiencing a DNS redirection/amplification attack occurring on port 53, with a lot of port 0 UDP packet fragments being generated as collateral traffic.

Is Something Underneath This Attack?

So far we’ve gotten a lot of insight into the details of the DDoS attack from full NetFlow details. But is this volumetric DDoS the main event, or are we being distracted from looking for other, less obvious threats? We can see a lot of packets being sent to port 4444 (green line in graph).

Port 4444 is the UDP port for the Kerberos service, and is—at least for Windows machines—a well-known target for buffer overflow attacks, often used to insert trojans such as Hlinic and Crackdown.

So, potentially there are two types of attacks going on in parallel: a DDoS attack and a buffer overflow trojan insertion. Many security blogs and publications note that DDoS attacks often are used to obfuscate other exploits. This very well may be an example of that technique.

Getting a Handle on Attack Mitigation

Characterizing the attacks leads us to mitigation. One way is to take the attack traffic and group it by x/24 source network addresses:

We now can take two mitigation steps:
- Ask our upstream ISP to drop this traffic or send it to a scrubbing service or device, if we have one.
- As an added precaution, drop all traffic from these countries going to port 4444 on our own routers.
Conclusion

One of the benefits of being able to dig into full-resolution NetFlow data is that you can get operationally useful insights without needing in-line devices. You also get more freedom to employ a portfolio of mitigation methods. Big data is the way forward if you want to have full details at your disposal to deal with DDoS attacks.

About the Author / Alex Henthorn-Iwane

Alex Henthorn-Iwane is the Vice President of Marketing at Kentik. He has more than 20 years of experience bringing new technologies in networking, security and software to global markets. Henthorn-Iwane leads the global marketing strategy for Kentik and help the company bring its innovative story and solutions to network-dependent organizations around the world. Connect with him on LinkedIn, Twitter or SlideShare.
August 5, 2016
The DevOps Force Multiplier: Competitive Advantage + Security

Several years ago, DevOps started hitting the eyes of enterprise business leaders when news outlets such as The Wall Street Journal and Forbes began reporting that the IT culture and methodology was helping adopters to gain a competitive advantage. Since then, we’ve seen story after story about how DevOps-focused employees are becoming the driving force behind the software and applications the world relies on for everything from mobile banking transactions to ride-hailing to gaming. Even the Pokemon company can’t get enough DevOps engineers.

It looks as if DevOps soon may be recognized as more than just a force for fast and agile software development and deployment. Evidence is emerging that it also may be having a positive impact on security.

Security Researcher Sees Trend

Bugcrowd Security Researcher OJ Reeves

Bugcrowd Researcher OJ Reeves sees his fair share of weak security. As a full-time white-hat penetration tester and researcher running attack simulations for enterprises that want to improve their defenses, he’s seen everything from unpatched vulnerabilities in software to the perilous journeys overshared credentials take as they’re passed between multiple administrators across various communications channels.

Recently, Reeves noticed that in our facepalm-ridden world of security blunders, organizations with a DevOps focus seem to have better security.

“DevOps is young and still being defined differently across organizations,” said Reeves. “I have noticed recently that when I engage with businesses that employ DevOps-focused people, a security mindset often accompanies.”

The reasons behind why DevOps is assisting with creating more secure environments are wide and varied, Reeves cited several.

DevOps Checks Boxes — All of Them

“Check box security” can be a bit of a dirty-word term when it comes to protecting systems and data, as it frequently refers to compliance—something many cybersecurity and risk professionals consider to be the bare minimum when it comes to protecting data and systems. At the intersection of DevOps and security, it actually can mean something very different, Reeves said.

“Organizations that actually have DevOps people in place tend to do a better job checking all boxes and making sure security is in place,” he said. “When you have a DevOps person who is doing the job properly, they (organizations) have a much better chance of defending themselves.”

DevOps Makes Fewer Mistakes

One company Reeves attacked had developers, operations and infrastructure employees in place but no DevOps-focused team members. He noticed right away that there was no clear definition of who was responsible for security or who was in charge of protecting application life cycles.

“There were credentials everywhere—all spread out and all mismanaged in different ways,” he noted. “You don’t want anyone with malicious intent, who can compromise a developer’s’ desktop, to get access to credentials.”

When running simulations for organizations with DevOps employees in place, he’s observed that those kinds of mistakes happen less frequently.

“More often than not I’ve found that the credential management process is much better when DevOps people were present. When there is a person in charge, who is also conscious of how others are using credentials, a credential security champion tends to emerge,” he said.

DevOps Communicates

Reeves, like many other practitioners I’ve spoken with, emphasized that communication between IT members is critical to effective defense. That’s something he said DevOps-focused organizations seem to be especially good at doing.

When privy to conversations taking place between developer, security and DevOps employees, Reeves has noticed that everyone involved is interested in improving processes all around—processes which include security.

“Direct human interaction that takes place between those who are responsible for defense tends to result in improved security,” he added. “Just the fact that awareness for these [security] issues is being raised and discussions are taking place is having a positive impact.”

Reeves indicated the discussions taking place within DevOps environments he’s observed are focused on the right things.

“These human-to-human interactions make things more secure and they pay attention to organizational security, where we should be having our discussions,” he noted. “Let’s not talk about the latest AV (anti-virus), etc.; security is an organizational process and problem. The best security in the world doesn’t matter if employees in the organization aren’t aware of what’s taking place.”

DevOps: Harder to Crack

At the end of the day, companies such as Bugcrowd and pen testers such as Reeves are paid to find exploitable gaps in systems so they can be closed before the nefarious can take advantage of them. While DevOps isn’t typically thought of as a security enabler, organizations that don’t have dedicated DevOps employees may want to consider Reeves’ parting words: “When organizations I am researching have a DevOps focus, it is generally more difficult for me to compromise them.”

July 28, 2016
How DevOps Can Help Improve Security
Automation and standardization along with development-inspired reviews can address the three top risks to IT security.

The term “DevOps” these days tends to evoke images of automation and push-button application deployments whenever app dev wants. It’s anarchy, it’s chaos, and it’s a frightening notion to those for whom stability and security of the core business network is their top priority. After all, folks in the DevOps camp routinely cheer on a “Chaos Monkey,” whose sole purpose is to break things in the production network. On the surface, that hardly seems conducive to stability or security.

So, it might be surprising to discover that the seemingly laissez faire approach to building and delivering apps with DevOps is actually one of the best ways to combat significant risks to IT security.

Not kidding.

First, let’s consider the survey, conducted by SpiceWorks, in which IT pros were asked to rank a set of threats in order of risk to IT security.

According to the report, the respondents ranked the following threats as their organization’s biggest three risks to IT security as human error, lack of process and external threats.

All three of which DevOps can positively impact, without negatively impacting stability or reliability of the core, business network. Let’s examine how, shall we?

Human Error

We’ve all fat-fingered configurations and code before. Usually we catch them, but once in a while they sneak into production and wreak havoc on security. A number of “big names” have been caught in this situation, where a simple typo introduced a security risk. Often these occur because we’re so familiar with what we’re typing that we see what we expect to see, rather than what we actually typed. We’ve entered that command a hundred times, after all; we know what it’s supposed to look like and we aren’t as careful as we (likely) should be because it’s Friday and it’s almost 5 o’clock. Whatever the reason, we’ve done it, and it’s the repetitious nature of the beast that actually ends up biting us in the proverbial derriere.

Codifying those commands into scripts or, even better, into templates means once it’s right, it’s right. The inescapable reality that we will fat-finger something eventually is reduced to virtually nil because we aren’t typing it in over and over and over, each time with less and less attention. The reliance on codification, on templates and APIs and scripts as a means to automate tasks, is not just about speed but also repeatability. And accurate repeatability at that. That’s the kind of assurance you need to prevent “human error” from exposing apps—and the business—to a wide range of risks.

To reduce risk from human error via DevOps you can:

Use templates to standardize common service configurations;

Automate common tasks to avoid simple typographical errors; and

Read twice, execute once.

Lack of Process

This is probably my favorite risk (and not just because it lets me talk up Six Sigma and some really hairy mathematical equations) because it encompasses such a broad category of problems. First, there’s the fact that there’s almost no review of the scripts that folks already use to configure, change, shut down and start up services across the production network. Don’t let anyone tell you they don’t use scripts to eliminate the yak shaving that exists in networking and infrastructure, too. They do. But they aren’t necessarily reviewed, they are certainly aren’t versioned like the code artifacts they are, and they rarely are reused. Everyone has their “own” favorite language and scripts, and they are the Frank Sinatra of IT: They do it their own way. (Yes, I know you’re probably too young to get that reference, but until your musical icons have a better one, that’s the one I’m using because my generation didn’t have a better one, either.)

The other problem is simply there’s no governed process. It’s tribal knowledge. Bob does X and then Alice pushes Y and someone from Team Rocket* pushes button to make it go live. What can happen—and sometimes does—is one step is simply overlooked. It isn’t that there is no process, it’s that there’s no real standardization and no overarching governing process. There’s no one coordinating between Bob and Alice and Team Rocket to ensure the process goes smoothly. The thing is that the app might go live and actually work in production minus a few services, but if those services are related to security in any way (such as locking down a port), then your lackadaisical approach to the deployment process just put the business at risk.

To reduce risk from lack of process:

Define the deployment process, clearly. Understand prerequisites and dependencies and eliminate redundancies or unnecessary steps.

Move toward the use of orchestration as the ultimate executor of the deployment process, employing manual steps only when necessary.

Review and manage any scripts used to assist in the process.

External Threats

At first glance this one seems to be the least likely candidate for being addressed with DevOps. Given that malware and multi-layered DDoS attacks are the most existential threats to business today, that’s understandable. There are entire classes of vulnerabilities that only can be detected manually, by developers or experts reviewing the code. There’s DevOps in that, but it doesn’t really extend to production, where risks becomes reality when it’s exploited.

More extensive testing, and development of web app security policies during development that can then be deployed in production is one way that DevOps can reduce risk. Adopting a DevOps approach to developing those policies—and treating them like code, too—provides a faster, likely more thorough policy that does a better job overall of preventing the existential threats from being all-too-real nightmares.

To reduce the risk of threats becoming reality:

Shift web app security policy development and testing left, into the app dev life cycle.

Treat web app security policies like code. Review and standardize.

Test often, even in production. Automate using technology such as dynamic application security testing (DAST) and when possible, integrate results into dev life cycle for faster remediation that reduces risk earlier.

There is no technology, methodology or approach to IT security that will completely eliminate risk short of a complete power outage. Given that’s not a viable option, the best approach is one that reduces risk to acceptable levels (where acceptable is defined by your business’s tolerance, regulatory requirements and how much your MBO is based on not being hacked this quarter). DevOps and security are a good fit, as the former can help standardize, codify, automate and expand security practices across production and dev/test.

So get out there, and put some DevOps in your security. And while you’re at it, put security in your DevOps.

* Yes, that’s a Pokémon reference. I figured I needed something more modern after tossing Frank Sinatra in there. If you got both, give yourself +5 geek points.
July 21, 2016
Does Security Slow Down DevOps?
Since the primary goal of DevOps is to eliminate bottlenecks to increase a company’s speed and efficiency, organizations tend to embrace new strategies that make DevOps even more efficient. However, when it comes to integrating security into DevOps, there is some debate whether such integrations help to improve the success rate of the DevOps process.

The Security Dilemma

Some arguments against integrating security into DevOps include:
- Security and DevOps function better separately,
- There is no need for security in DevOps, or
- Security will somehow slow down the creativity driven by DevOps.
Some say security integrated with DevOps stifles the work flow and innovation due to its restrictions. In other words, security can viewed as getting in the way of development, or being overbearing to the process, because it is not geared toward rapid development, but rather toward safety.

Another Perspective

Despite the arguments against integrating security with DevOps, security breaches because of neglecting safety during development can be a nightmare. For example, software or a website attacked via an overlooked vulnerability point, or a major bug to the code, costs the company in terms of time, money and competitive edge.

Not only can it crash the product, but it also can halt sales or seriously damage the reputation of the product or site. If security is not integrated with the fast-paced DevOps team, security problems often are not solved or even detected until well after the product is released. This can make it very difficult for the security team to limit the negative effects of any security problem.

The Benefits of Integration

Security integrated with DevOps creates an opportunity for the security team to fix and identify problems or vulnerabilities before the product is launched—and before the stakes are suddenly much higher and much more costly.

Integrating security into DevOps allows the security team to protect the code and the products from within, while they are being created.

It can be compared to safety inspectors being part of a team that builds a railroad: Wouldn’t it be easier for them to identify any problems with the railroad while it is being built, rather than trying to fix them after a train crashes?

Security from the Beginning

It is true that DevOps and agile development involve being very adaptable to change and making rapid prototypes and new products that account for errors, flaws or even security risks. However, just because a DevOps team can move fast doesn’t mean the business won’t be more profitable if it can prevent security risks in the first place.

Developers don’t possess the same knowledge and skills security professionals have. For that reason security professionals should be included in the development process to increase the chance of releasing a stronger and more effective product.

The database specifically is an area that can spell disaster. Security and regulatory compliance should be a central component of DevOps for database to prevent unauthorized and undocumented changes to the database and a potentially disastrous security breach.

It’s important to protect a developer’s creativity and workflow, but most companies have room to integrate more security into their DevOps team. Doing so might just save them from huge problems down the line.

Next Steps

Here are a few key steps to enable secure DevOps:
1. Configure the dev, test and deployment environments identically.
2. Make proactive changes to all environments automatically, significantly reducing the opportunity for engineers to make security mistakes.
3. Create a secure process and determine roles and responsibilities as early as possible in development stage.
4. Perform all vital secure connectivity reviews during the development process.
5. Implement separation of duties across your entire release process.
6. Fully automate your deployments to reduce the need for manual access and attended processes.
7. Ensure your database does not become a compliance and security risk.
About the Author/Yaniv Yehuda

Yaniv Yehuda is the co-founder and CTO of DBmaestro, an enterprise software development company focusing on database development and deployment technologies. Yaniv is also the co-founder and the head of development for Extreme Technology, an IT service provider for the Israeli market. Yaniv was a captain in Mamram, the Israel Defense Forces computer centers, where he served as a software engineering manager.
June 30, 2016
DevOps Security in Spotlight as Gartner Names vArmour Cool Vendor

News Reports vArmour CEO Eades Nabbed $41 Million Ahead of Trump Presidency

In May, Gartner named 2016 Cool Vendors in several security categories. Given the focus of this column, I was interested in several of the companies chosen for the designation, especially those named in the Cool Vendors in Cloud and Emerging Technology Security 2016 report. Interestingly enough, only vArmour, one of the new-era cloud security and visibility providers, was noted for supporting security in DevOps environments.

Click to Tweet: Cloud Security #CoolVendor @varmour noted by @Gartner for #DevSecOps, datacenter vis. The latest @devopsdotcom: http://ctt.ec/uSRLd+

In the report, analysts Joerg Fritsch and Brian Lowans wrote:

Supporting the vision of a software-defined data center, all of the capabilities of the vArmour platform are exposed and consumable programmatically via APIs, providing support for highly automated and DevSecOps-type development workflows.

The analysts also noted vArmour enables application-aware microsegmentation anywhere, from the data center to the cloud and all hybrid points in between. They also highlighted its capabilities that extend security to virtualized environments and containers—security anytime and all the time. Pretty cool, indeed.

The report provides deep validation for a message vArmour has been pushing since its emergence from stealth—visibility. While many security vendors have interlaced the keyword into their marketing content, this report makes the claim for vArmour all the more true. According to the report:

vArmour’s distributed security platform combines a scalable fabric with advanced security analytics that offer full visibility from a deep packet inspection engine. This deep visibility enables other cool use cases, such as network-based threat detection and the visualization of workload traffic. This is required in environments where the workload, the network and the security policies are interacting much more closely than we have been used to in traditional environments.

Validation from firms such as Gartner can carry heavy weight among buyers, especially among deep-pocketed enterprise influencers. I’ve had conversations with C-level security execs at numerous mega corporations in the past who have stated point-blank that they won’t approve expenditures for any technologies that aren’t noted in recognized industry reports.

vArmour also announced recently the closing of a $41 million round led by Redline Capital and Telstra. This D round lifts the software-defined security provider’s total investment to $83 million.

In the official company release, vArmour stated it will use the capital for “global expansion and to accelerate worldwide software distribution of its Distributed Security System (DSS) through strategic partners in Asia-Pacific, EMEA and in North America.”

Although the Presidential race wasn’t mentioned in the release, it was reported by CNBC’s Ari Levy that the real reason CEO Tim Eades sought funding now was in anticipation of a Donald Trump presidency. Eades was quoted in the CNBC story as having said:

It really does matter what people think of the U.S. as we lead up to the election,” said Eades, a native of the U.K. who moved to the U.S. 18 years ago. With Trump, “the conversation he’s had around Mexico, trading abroad, his position on immigration — everybody outside America looks in and says, `What’s going to happen?’

Levy continued:

With Trump having closed the gap on Clinton in the polls less than six months before the election, Eades said it’s definitely time for start-ups to be padding their cash positions. He isn’t comforted by the fact that Trump is a businessman himself.

To most reading this, vArmour being named a Cool Vendor and handily raising $41 million likely isn’t coming as a surprise. Starting with Eades, the company has an impressive cybersecurity and business leadership pedigree, especially when it comes to startups that are shaking up security thought and practices around the data center, cloud and DevOps movement. Eades is even viewed as a thought leader—likely one of the reasons why he is an occasional guest on global news shows such as Fox and Bloomberg, where he has been called to provide expert analysis on everything from cyberwar to breaches to investment.

His vision to extend security to the rapidly growing trends of DevOps, cloud, containers and automation is certainly smart, as these are directions most enterprise IT organizations are headed. Said Eades in a company issued release on the funding:

It has been an amazing 18 months since we launched our company. We now have hundreds of customers adopting the vArmour Distributed Security System on a global scale, including some of the world’s largest telecom companies, banks, healthcare, retailers and U.S. government agencies. We believe Gartner’s ‘Cool Vendor’ recognition further validates the positive impact and true value we’re providing our customers. We’re offering them the industry’s simplest, integrated data center and cloud security solution to prevent, detect and remediate advanced persistent threats inside their network.

June 17, 2016
Lean Security: How Better Development Can Protect Your Business

If companies are to reach their strategic goals—reducing time to market, boosting sales, improving product market fit and brand image, and cutting cybersecurity costs—then it’s time for a new outlook on software security.

Today’s business leaders must learn to see security for what it is: A differentiating factor. Companies with reputations for secure developmental processes and infrastructure will rise above those known for data breaches.

Security as a Business Strategy

The road to a more secure company—and realizing the perks associated with it—begins with lean security. This is an approach to information security similar to the Toyota principles of management and production that calls for environmentally aware engineering, simplified coding, automation of security checks and constant incorporation of feedback.

Although some business leaders believe organizational security is inherently expensive, lean security doesn’t mean massive costs. Much like DevOps, lean security champions process improvements and cultural changes above purchasing new tools.

Lean security does, however, require engineers to keep its principles in mind throughout the development process. In this way, security is much like sales: A sales representatives’ best strategy is to consider his goals from the beginning. Just as the salesperson isn’t trying to make any sale, the engineer isn’t trying to create just any product. For the salesperson, this means tweaking the sales process—perhaps by vetting leads before engaging them—to make profitable sales. For the software engineer, it means structuring the development process through lean security to create secure products.

Done right, lean security results in a self-defending, simple system created by people who are aware of a product’s real-world security risks and the company’s business goals. Lean security recognizes your company’s data is always at risk, no matter the size of the company or team, and it works to create value through systemic protection. Here’s how to start:

Improve Environmental Awareness

Lean security approaches risk management as a team effort. Think about a hospital. Everybody working there — from the cooks to the nurses to the executives — has a responsibility to take care of patients.

Software firms must function the same way: Everyone from engineers to salespeople to the CIO must keep data privacy, software security and business needs in mind throughout product development and a product’s life cycle.

Environmental awareness means addressing security with the same sensibilities that you would your own life. Just as you don’t hand your credit card to random people, neither should you hand the keys to a software engineer who doesn’t know the risks of insecure development.

More secure coding begins with a change of mindset. Once team members understand a product’s security risks, they start thinking about how to create a more secure environment. If a developer is creating a mobile app for a car, for instance, then they must recognize the app will be used in conjunction with a device that carries human life.

Simplify Engineering

Complexity is the enemy of security. Simplification, however, cannot just be limited to lines of code: It means organizing your teams—and thus, your entire system—in a way that improves time to market and tightens feedback loops.

DevOps is a great way that business leaders can improve security through sensible reorganization. While the integration of development and operations teams has been lauded as a security boosting measure by software leaders at companies including Dell, CA Technologies recently studied DevOps’ business benefits. It found companies that have implemented DevOps techniques are 2.5 times more likely to improve customer retention, twice as likely to grow their revenues and 3.4 times more likely to improve their market share.

In short, keep teams compact and agile and design with the end in mind. The more lines of code you add, the more complex teams you create and the more times you reinvent the wheel—and, ultimately, the less secure and less profitable your company’s products become.

Automate or Die

Automation is an essential component of lean security, and DevOps engineers are no strangers to automation. Threats are constantly increasing, and no engineer—or team of engineers—can discover and remedy them all manually.

Look to Netflix for a great example of automated security. Once upon a time, Netflix addressed malware alerts by manually creating a help desk ticket and assigning an engineer to investigate the problem. The time from alert generation to eventual resolution, according to Netflix, often spanned more than a week. As Netflix’s security challenges increased in both diversity and complexity, it found itself spending exponentially more time and money combating these threats.

Then, Robert Fry, Netflix’s senior information security architect, came up with a brilliant solution: He engineered FIDO, an orchestration layer that automatically evaluates, assesses and responds to security threats. FIDO detects threats, then scores them based on the attack’s intended target and other factors. FIDO then attempts to mitigate the event by closing a network port, ending a VPN session or disabling the account.

Tools such as Netflix’s FIDO automatically detect risks in seconds, while it could take a human months to stumble across them. Think of security automation like a home security system: ongoing, vigilant security without the need for continued manpower.

Measure Everything

Although its origins are disputed, many credit English mathematician Karl Pearson with quipping, “That which is measured improves. That which is measured and reported improves exponentially.”

Development teams should take Pearson’s wisdom to heart, and measurement must begin with identification of goals. Is your goal to reduce application downtime or decrease event response time? Then it’s essential to measure mean time to repair. Do you want to reduce code defect density? Begin by measuring the number of issues per thousand (or million) lines of code.

Once you’ve identified goals, use software to track progress over time. Etsy’s engineers built a tool they call StatsD. The software helps Etsy monitor everything from login failures to coffee availability. The data is then displayed in handy graphs to help the team make sense of the information.

Once you’ve taken baseline measurements, be sure to review your metrics on a regular basis. For instance, you might realize an application’s deployment times aren’t where you’d like them to be. Keep measuring results—and keep trying new things—to nudge those values into better territory. Just remember: You cannot improve that which you do not measure.

The Business of Lean Security

The value of a secure, simple system cannot be understated: It means getting products to market faster; it means better public perception and press; it means more satisfied customers who feel secure using your product; and it means fewer disruptions in engineers’ work schedules. At its core, lean security lowers a company’s overall exposure to risk and reduces its expenses.

Companies are already staking their futures—and consumers’ safety—on code that facilitates transactions, drives vehicles and manages power plants. The Internet of Things will soon run everything from toasters to jet engines, and security will grow even more important. Protect your company from within through lean security.

June 3, 2016
Hello, Security. Meet Agile
Illumio Chief Commercial Officer Alan S. Cohen recently wrote in his blog, “Go, Slow and No: Bringing DevOps Speed to IT Security,” “There are reasons why security and risk professionals often react with a ‘no.’ When you are tasked with assessing and reducing risk, going fast might not be the first instinct.”

Many InfoSec practitioners adhere to accreditation and certification programs such as ISO27001, SSAE-16 SOC and PCI compliance. This ultimately means they follow strict policies and procedures for software development. These procedures go through many gates, including risk analysis, design and code reviews, penetration testing and ITIL-based change-management processes. InfoSec teams live in a world where they have to plan their changes months in advance, and this world often is in direct conflict with agile development and DevOps concepts.

Agile is all about focusing on flow and end-to-end cycle time, releasing very small changes fast, early and often. It’s not about the number of things in each sprint, but rather what’s needed now for MVP and iterating. For agile and security to be in alignment, there needs to be a fundamental shift in the principles and values that enable the business over governance and process.

In particular, security:
- Embraces some uncertainty. You will never build a perfect piece of software. The key is to build guardrails and to have the required adaptive enforcement and monitoring in place to capture and correlate metrics that help you quickly respond, evaluate factors of compromise and leverage your deployment automation to remedy risk exposures quickly.
- Is part of the project team, either embedded or as an adviser. There must be a dedicated resource that understands the team’s mission of delivering value. Somebody on the team should be wearing the security hat, helping to guide each project.
- Is a risk adviser inside the organization. Don’t compartmentalize information. There needs to be a culture of trust, sharing and transparency about potential vulnerabilities to the service.
- Trusts the decision-making of engineering teams. Informed and capable engineers aren’t reckless with security decisions.
- Is part of everything. It’s the design of the entire service and therefore is the responsibility of the entire team, not just the security team.
- Improvements do not add complexity to the user experience. Security that degrades the user experience will force people into insecure practices.
Bringing Agility, Security Together

Embracing agility and DevOps opens the door to new possibilities that can improve security posture without hindering the business. Security will quickly discover a new set of best practices:
- Look to microsegmentation software that is adaptive and intelligent, to replace manual management of firewall rules and policy.
- Decouple security from the infrastructure to gain a continuous enforcement pipeline that adapts to new hardware, with strong versioning that proceeds in lockstep with product releases, monitoring and infrastructure configuration.
- Build security into the service stack; don’t bolt it on afterward.
- For change auditing, don’t rely on the change management database alone. This is highly prone to error and discrepancies. Look at the the git history integrated with the automated deployment system. Change management databases should be integrated with the git pull requests and commit logs. Infrastructure automation tools such as Chef and Puppet keep logs for unapproved change drift. Make sure those logs are being captured and reported on.
- In the area of developer support, DevOps is enabling the self-service/self-responsibility revolution by building wrapper abstraction layers for engineering teams to consume infrastructure across public and private data centers, using interfaces such as chat to trigger administrative actions and query metadata information on their instances. These wrappers can also be harnessed to present security grading for every server instance, promoting awareness and corrective action to each individual owner. You can quantify the grade by looking at patch vulnerabilities, images used to build, inbound/outbound port rules and configuration standards.
- For data at rest, data in motion and data in transit, everything should be encrypted, everywhere. For example, use Illumio SecureConnect to encrypt data in transit via IPsec between workloads. Automation and the right secrets-management infrastructure can enable frequent rotating of SSL certs and keys.
- Every time you commit to the code base, run a set of very basic security tests via your favorite continuous integration tool. Use automated tests to ensure common security mistakes don’t leak to production.
- Perform red team exercises as an attempt to gain access to a system by any means necessary. Try to mimic the same processes that a motivated attacker would follow to map out an organization’s infrastructure, perform reconnaissance at key physical installations, and then test the physical, cyber and social defenses all at once through a staged exercise.
Security can become an enabler. It no longer has to be stuck being the department of “No.” InfoSec can now be the team-building tools and systems that enable development teams to do what they want in a secure way.

About the Author / Alfred Garcia

Alfred Garcia oversees DevOps for Illumio, where he is responsible for all preproduction and production cloud/infrastructure engineering and site reliability. A technology and operations executive with more than 18 years of experience, Alfred is an aggressive advocate of DevOps and of deep operations and engineering collaboration, from design to code/test and release to production support.
May 9, 2016
Murphy’s DevOps: The Internet Apocalypse

I’ve been immersed in cybersecurity for a good stretch, since it was known as just “infosec” and “security.” Over time, I’ve had opportunities to work with some brilliant minds and young-gun “hackers” (I’m not a name dropper, so I’ll refrain from using them as SEO-bait). I’ve watched attackers transition from being basement-dwelling teenagers to organized intruders to APTs. At one point or another, I’ve met all of Five Horsemen of the Internet Apocalypse: Scam, Extortion, Embarrassment, Theft and Death.

The biggest cybersecurity news story of all time is still Edward Snowden. Fancying himself a “whistleblower,” champion of the Constitution and privacy rights expert, this fugitive data thief accused of espionage by the United States government has done more to force businesses and individuals to think about digital security and privacy than anyone or incident before him.

Snowden aside, there are thousands of other stories about just how fragile our online lives and businesses are. If you were at the devopsy.kinsta.cloud Rugged DevOps event at RSA 2015 and caught the presentation by Wired cybersecurity and privacy writer Kim Zetter, “2015 — The Year in Hacks,” then you had a chance to get a great overview of some of 2015’s biggest digital crimes. You might have even noticed the Internet Apocalypse Horsemen riding through.

With an ear to the ground on everything the NSA and APT-using hackers are guilty of, @KimZetter never fails to intrigue readers and live audiences. Whenever you read her stories or listen to her speak, you always walk away reassured that when all the smoke and headlines have cleared, the great deep-down reality is that everything we digitize is subject to the whim of Scam, Extortion, Embarrassment, Theft and Death. If you didn’t have a chance to attend, please enjoy some of the highlights listed below.

China vs. U.S., Spy vs. Spy

She kicked off with the biggest hack of 2015: the one that penetrated the U.S. Office of Personnel Management (OPM). The full extent of the damage done and who the actual attackers were may never be fully realized. Were they Chinese spies, rogue hackers or a combination of both? What we do know is that it exposed millions of identities tied to covert U.S. operatives and their family members listed on security clearance forms. It may have even exposed personal information on everyone who has ever applied for a job with the federal government. One for-sure lesson is that even after several decades of security product development (with more than 800 available for review at RSA), the world is still in need of innovation in the space.

Men Behaving Badly …

Some might argue that Embarrassment made a special appearance at Ashley Madison, but the fact that suicide was part of that hack leaves me to think that Death was the lead actor there. @KimZetter pegged the data theft as the most brazen hack of 2015. With 30+GB of account information stolen and dumped, login credentials for 32 million accounts purloined, credit card transaction information accessed and even several suicides tied to the incident, the hack again reminded the world that the great deep-down reality is that everything we digitize can be used by attackers to scam, extort, embarrass, steal and even kill us.

From the Keyboards of Babes …

Anyone at the presentation must agree that Embarrassment really stole the show in the hack of CIA Director John Brennan’s AOL email account. Allegedly perpetrated by a teenager, Zetter describes the incident more deeply in her October story: Teen Who Hacked CIA Director’s Email Tells How He Did It

Keep Calm and Internet On

Anyone reading this may be walking away with a rather pessimistic view of our chances of remaining secure online. The truth, however, is that the existence of determined hackers should do no more to stop individuals or businesses from going online than road rage should to prevent anyone from getting behind the wheel of a car.

If any lesson can be learned from news articles pumped out by @KimZetter and her peers, it’s that the world should continue to take full advantage of the Internet but to also take the time needed to apply effective security. Quite simply, we all should keep calm and Internet on …

April 22, 2016