devopsy.kinsta.cloud

Category: DevSecOps

Moving Security To the Left In a DevOps World

Moving security to the left has become a coined phrase meant to describe the process of getting the security team involved earlier in a process. Most typically, the phrase is used in conjunction with IT or software development projects. One of the top suggestions for ensuring security in a DevOps world is to move security to the left in the process tool chain. But what exactly or how exactly can you move security to the left?

Grab that Open Seat

The new DevOps process pipeline created an opening for a seat at the table for security. Prior to DevOps, the development team owned the entire pipeline from plan to release. The Ops team would historically receive the release from over the wall and then be responsible for the deployment and operating the software inside the production environment. With the Ops team joining forces with Development in the process pipeline, that has opened a proverbial seat at the table. What I’ve started to witness are security teams taking advantage of the musical chairs to quickly grab the open spot.

Taking a seat at the end of the development pipeline is not enough. Too many security teams are still being relegated to positions too late in the process. While running pen tests and security code reviews after a release is better then nothing, its still not ideal. Security needs to find ways to add value to the process so they can act as a force for positive change instead of the after thought.

Threat Vector Analysis

Threat vector analysis is part science and part art and part trying to guess uncertainty. What is certain, however, is that most developers would rather be spending their time writing new functionality instead of trying to understand how an attacker could be break their code. The security team could offer a tremendous service to an organization by offering their expertise in this area.

Continuous Integration Security Testing

Continuous integration is a key component of DevOps. The security team can leverage the development chain to insert early yet important risk control tests. Those tests could be for example static or dynamic code analysis. Other tests could be simple yet known vulnerability checks in included libraries. The OWASP introduction to testing guide lists a plethora of application security tests that should be checked often and early.

Author Micro Security Services

Think about how great it would be if there was a group who specialized in authoring and delivering security related services into your application? Proponents of services models generally promote splitting the development team up into groups that are organized around business capabilities. For example, there are separate development groups for UI, middleware and DB. What’s missing in most of those models is who is responsible for security services such as authentication, authorization and audit. Security has an opportunity just waiting for them to add incredible value while also ensuring the security services used pass muster.

Summary

Where we once had Ops fighting their way to be part of the entire development process, we now have security trying to do the same. Ops managed to garner their way into the club by coming to the table with invaluable skills and resources. Now its up to security to present their creative sweet spot in order to secure their own seat at the table.

Further Reading

Andi Mann on Ensuring security and managing risk in enterprise DevOps

October 1, 2014
ShellShock vulnerability: Impact, analysis and protection

This week Stephane Chazelas, an Akamai security researcher, discovered an interesting bug in the Unix Bash (Bourne Again Shell) shell – known as “Shellshock” or “Bash Bug”. This vulnerability affects a wide range of operating systems (Unix, Linux, OS X), routers, and other technologies, making many of the systems we use on a daily basis vulnerable to attacks. A system vulnerability in the GNU Bourne Again Shell (Bash), the command-line shell used in many Linux and Unix operating systems, could leave systems running those operating systems open to attack. The bug utilizes the Bash process variables that are passed by the operating system or an application. If Bash is configured as the default system shell, it can be used to attack devices via web requests, secure shell, telnet sessions, or other programs that use Bash to execute scripts. Patches for this vulnerability are now available for all the major Linux distribution vendors.

THE IMPACT
The reason this vulnerability raises many concerns is the breadth in threat vectors available to attackers. For example, the simple task of loading a website gives attackers the opportunity to exploit the vulnerability if the server handling your website request uses bash commands to access the information you requested. So if you still don’t think we use command line anymore, think again.

In fact, there are some computer programs that use Bash to execute commands on other computers. For example, how does a website know if you are accessing the site via mobile device or computer web browser? Based on the response your device returns ,the server will modify the content delivery to be compatible with the corresponding platform. From a technical perspective, your browser sends an agent to tell the web server that you are using either a mobile device or computer browser. The web server then uses that information to set the agent string as a variable before using Bash to execute code that generates the web page.

Attackers are making use of this vulnerability as we speak. Several researchers are reporting that honeypot infrastructures are being actively attacked by a group exploiting the Bash vulnerability. The vulnerability is allowing the installation of a zero day bash injected ELF malware that is currently being used. A system administrator using the @yinettesys Twitter account published a GitHub post reporting that threat actors exploited the Bash Bug flaw to launch kernel exploits on a computer system. The malicious code then performs a Web GET request from a user agent to begin a brute force dictionary attack to harvest credentials. The malicious code for this attack has been analyzed by virustotal and at this time there is 0/55 detection by antivirus.

PROTECTING YOURSELF
There are several detection methods that can be used to mitigate this risk. Since the Bash Bug affects versions 1.14 through 4.3 of the Bash shell, upgrading to the latest version will mitigate the risk.

From a web application standpoint, the bug can be exploited by code that passes through the Bash interpreter. CGI’s (Common Gateway Interface) and CGI scripts are the most affected, but anything that is passed to the Bash interpreter can be exploited. Command execution can be achieved through HTTP Headers and POST/GET parameters on vulnerable systems. So you could use a web application firewall to monitor the vulnerability in the header. Additionally, a signature will be added to the POST/GET field. This signature will monitor for attempts to bypass the detection signature via multiple whitespace using ( ){ command and ( ) { command and ( ) {.

Use an IDS/IPS to detect for any type of network communication; IDS/IPS will notify you when a connection is established and remote commands are executed.

We will continue to monitor the situation and post any new information we uncover to the Alert Logic blog.

SEE IF YOU ARE VULNERABLE

To see if you are vulnerable to this exploit run the commands below, the response will tell you if you are vulnerable

env x='() { :;}; echo vulnerable’ bash -c “echo this is a test”

If the system is vulnerable, the output will be:

vulnerable this is a test

An unaffected (or patched) system will output:

bash: warning: x: ignoring function definition attempt bash: error importing function definition for `x’ this is a test

CREDITS
https://lists.debian.org/debian-security-announce/2014/msg00220.html

http://blog.erratasec.com/2014/09/bash-shellshock-scan-of-internet.html#.VCRpVitdXKN

http://arstechnica.com/security/2014/09/bug-in-bash-shell-creates-big-security-hole-on-anything-with-nix-in-it/

https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2014-6271

——————————————————

Here are signatures for the IDS for detection that have been provided by Alert Logic.

These rules have been created with resource consumption (No PCRE) of everyday average hardware to have more comprehensive coverage of use.

We have to use “any” here because lots of services open web servers on non-standard http ports such as cups and devices any, because local attacks would be common ground for this (cups, avahi, etc.) and http_header likely only uses a whitelist we will miss unique header values. This can also be bypassed due to the “within” modifer, but we got to try to narrow this down for resource issues.

External net would be the outside Internet IP address. Home Net would be any communication from internal IP space.

alert tcp $EXTERNAL_NET [1024:] -> $HOME_NET any (msg:”AL: HTTP Server
Headers Bash Environment Escape RCE”; flow:to_server,established;
content:”() {“; http_header; fast_pattern; content:”}|3b|”; within:20;
http_header; tag:session,5,packets; classtype:attempted-admin;
sid:1006236; rev:1; )

Uri and method are theoretical for now, but check just in case another server does this; any input poorly parsed to an internal scripting function like Popen could trigger:

alert tcp $EXTERNAL_NET [1024:] -> $HOME_NET any (msg:”AL: HTTP Server
URI Bash Environment Escape RCE”; flow:to_server,established;
uricontent:”() {“; fast_pattern; uricontent:”}|3b|”; within:10;
classtype:attempted-admin; tag:session,5,packets; sid:1006237; rev:1;
)
() { should never be in a method, so we don’t need additional checks here:

alert tcp $EXTERNAL_NET [1024:] -> $HOME_NET any (msg:”AL: HTTP Server
Method Bash Environment Escape RCE”; flow:to_server,established;
content:”() {“; http_method; classtype:attempted-admin;
tag:session,5,packets; sid:1006238; rev:1; )

POST values might get parsed by the end application into environment variables (parsers, interpreters, formats, etc.) making one just to be safe, this might false, though

alert tcp $EXTERNAL_NET [1024:] -> $HOME_NET any (msg:”AL: HTTP Server
POST Bash Environment Escape RCE”; flow:to_server,established;
content:”() {“; http_client_body; fast_pattern; content:”}|3b|”;
within:20; http_client_body; tag:session,5,packets;
classtype:attempted-admin; sid:1006239; rev:1; )

Now the big bad one. An uploaded file could have metadata parsed and stuffed into an env variable (cups is notorious for this.) The optimization for this is going to be hard, but we have to do it so we don’t parse the max file parse byte:

alert tcp $EXTERNAL_NET [1024:] -> $HOME_NET any (msg:”AL: File
Transfer Parsing Bash Environment Escape RCE”;
flow:to_server,established; file_data; content:”() {“; within:100;
fast_pattern; content:”}|3b|”; within:20; tag:session,5,packets;
classtype:attempted-admin; sid:1006240; rev:1; )

Alright, now cookie values for base64 && unserialize/json_decode values that could be used. This could likely false due to these characters used often in these compressed data formats. This attempt is totally random because we don’t know the size and start of random people’s base64 encoded values + so rule might not even work, but we need to try disabled unless we need it due to the unknown and random chance of it working customer specific values where the cookie is known can easily be replaced in this rule:

#alert tcp $EXTERNAL_NET [1024:] -> $HOME_NET any (msg:”AL: HTTP
Cookie value Bash Environment Escape RCE”; flow:to_server,established;
content:”=”; http_cookie; base64_decode:relative; base64_data;
content:”() {“; tag:session,5,packets; classtype:attempted-admin;
sid:1006241; rev:1; )

DHCP sets user hooks

alert udp $HOME_NET [67,68] -> $HOME_NET any (msg:”AL: DHCP Bash
Environment Escape RCE”; flow:from_server,established; content:”() {“;
fast_pattern; content:”}|3b|”; within:20; tag:session,3,packets;
classtype:attempted-admin; sid:1006242; rev:1; )

This article was also posted to the Alert Logic blog

About the Author

Stephen Coty joined Alert Logic in 2012 as the director of threat research and author of the Cloud Security Report, and has a wealth of experience in the security space. In his role at Alert Logic, Stephen is currently in the process of building a premier threat research team that is innovating the industry by taking a more proactive versus reactive approach to security through solid research, reverse engineering of malware, and constant searches for stolen data, all of which can be leveraged as actionable data for remediation. Stephen has over 15 years of experience in systems engineering and security consulting, in industries ranging from healthcare to finance and government. Most recently, he held a senior position at Rackspace Hosting as the Manager of Cyber Security, where he served as the architect of the Security Operations Center and Threat Research Team. He is also a member of ISSA, Infragard and the HTCIA.

September 27, 2014
SecDevOps: the new black of IT

Much has been written about the role of Security in a DevOps culture. Almost universally the thought is that DevOps can help improve security.

Why?

How?

Really?

An old friend of mine from the security world Andrew Storms and I will discuss this and other issues around DevOps and security on June 11th at 1pm eastern time, 10am pacific time on a webinar entitled “SecDevOps: the new black of IT“.

Andrew is the Sr. Director of DevOps at CloudPassage. Prior to that Andrew headed up the security research team at nCircle. He is oft cited in the security press and highly respected. Andrew and I ill discuss how just when DevOps is catching on, adding to security to the mix can make it even more powerful. How can security help DevOps to succeed? How can DevOps help us with better security?

We will discuss what this means in terms of different kinds of security tools and services, the cloud and more.

If you are interested in security and DevOps register now and tune in to the webinar next Tuesday

June 5, 2014
Dev + Ops + ???
If there’s one thing the DevOps cultural and professional movement taught us, it’s to question the status quo for the betterment of our customers, businesses, and ultimately, our teams. The combination of improved tooling, questioning structural barriers between teams, and a “shared outcome” culture have been powerful forces to date.

Granted, we’ve seen the most demonstrated benefit in smaller organizations, but let’s face it… a big part of the movement is culture, and a set culture in a large organization is not easy to evolve. As we’ve mentioned before, you can’t sell DevOps. It will take time for large organizations to benefit from the movement.

While we wait for the cultural evolution of large organizations to DevOps, it’s worthwhile to watch the continued professional and cultural evolutions of smaller organizations. The ones that first embraced DevOps from their start, what’s next for them?

There are three ways I am currently exploring this question:
- Dev + Ops + ???: Addition of a third functional pillar
- DevOps evolution in parallel to Sales & Marketing evolution: Some interesting parallels
- Org-wide evolution: DevOps cultural tenets deployed throughout the organization
I would love to hear your observations as well.

Dev + Ops + ???: As the title suggests, the most common finding thus far is another legacy functional silo having the structural, cultural and tooling barriers to collaboration torn down. These organizations have a solidly executing group of engineering and operator mindsets working closely together, and have actively embraced a third pillar of the organization. The most common I’ve found is the security team and the embrace of the term SecOps, though I’d love to find others. Do you fit this profile? Leave a comment below on your observations.

.@cvwdyn What about barriers between engineering team and security pros? Need working contract to swiftly and safely integrate needs #SecOps

— Chris Gervais (@cgervais) June 3, 2014

DevOps evolution in parallel to Sales & Marketing evolution: In a small handful of organizations I’ve talked to, I’ve seen an interesting parallel evolution in the go-to-market teams to what was evolved in the technical teams.

If DevOps tore down the fence where code was tossed over, and eliminated the “blame game” when things went wrong of “It’s not my servers, it’s your code” and “It’s not my code, it’s your servers”, there’s a remarkably similar parallel evolution happening in go-to-market teams… of course not about code, but instead about leads. To put it the most bluntly ableit controversially, there has been a long-standing tension between marketing, historically charged with generating leads for new customers, and sales, historically charged with engaging the leads and nurturing them into prospects and ultimately happy customers.

Cue the blame game: “It’s not my leads, it’s your close” and “It’s not my close, it’s your leads”. Enabled by better tooling and visibility (e.g., customer relationship management software, marketing automation, etc.), progressive organizations are spending less time blaming each other, and more time collaborating on a shared outcome. Personally, I find the parallels between the evolution of technical teams and the evolution of go-to-market teams fascinating, and if you fit this profile, I’d love to hear your observations.

Org-wide evolution: Our preceding two profiles are fundamentally about change from within; specific cultural, structural and tooling barriers being torn down between functions of an organization. We see examples of technology teams tearing down walls, and examples of go-to-market teams tearing down walls, but what about the many other key functions of an organization?

Where’s finance? How about talent and human resources? Customer services and support? Product management? I would love to learn of more examples where cultural evolution is occuring not just in functional pockets of a comany, but across the entire organization at scale. If you fit this profile, I’d love to hear your observations.
June 3, 2014
Deputizing Everyone for Security – Building Agile Assurance

Those of us highly focused on the delivery pipeline of DevOps will wonder why we should include the security guy to the party. After all, aren’t they just going to slow down the process and make it harder to deliver good features to end customers?

My prior post about introducing SecDevOps by example was not about defining a new role, but introducing how security can and should be part of the automation delivery suite. Whether you are talking about traditional infrastructure change process or automated instantiation of new nodes to handle customer demand, the tools and processes of DevOps merged with security requirements are available to provide security accountability and assurance. Those of you coming from the traditional security space, which might fear security automation, should consider just how exciting SecDevOps delivers on agile assurance.

Rich Mogull provided an excellent example and code to show how firewall changes in a real life situation can be fully automated. There is no need to be squirrely about what is emerging. Everyone needs to be involved with development and operations to take more opportunities to minimize risk.

Dwaye Melancon also recently wrote about the issues of security and risk when the baton is handed off from one owner to another. This too is another great example of where process and monitoring automation aids in delivering continuous security assurance.

I believe everyone craves for a future when security events are automated with full transparency from beginning to end. If security had a real closed-loop transparent agile process, then just imagine how easy it could be to deliver on both compliance and real risk reduction. As Richard Stiennon recently stated in a CloudPassage webcast “Compliance standards are woefully behind with the reality of operations”. Haven’t we all had an experience with an auditor and a clipboard asking for specifics that are clearly aligned with another age of computing?

Wouldn’t it be better to give auditors the same tools, to get a picture of accountability and risk decisions made in the DevOps cloud environments? Take for example the recent HeartBleed events that many security teams are still struggling with. As a DevOps professional, I would have rather just showed the security team and the auditors the truth – that none of my team did anything at all to mitigate this risk. Truth be told, our orchestration tools had already installed the new version of OpenSSL and generated audit logs before many people had their first cup of coffee.

Security can be improved with more integrated automation and by deputizing everyone to do more with DevOps tools. A SecDevOps approach will improve the compliance and audit process with tactical security tools for all. Burn the clipboards and provide an enlightened way to prove best security practices from the collective group of Dev and Ops. Maybe give the Security guy a break and invite him to the after audit beer bash too.

May 21, 2014
Security in the operational relay race
If you’ve ever seen people run a relay race, you’ll notice that most of the risk happens when the baton is handed from one runner to the next. In IT processes, there is also dramatically increased risk during handoffs. Whether you’ve moving from one process phase to another, one “owner” to another, or one environment to another the risk is greatest during the phase transition.

Security can play a strong role during these handoffs, whether you’re using DevOps or not. Here are some principles that can help safeguard the handoffs and reduce the risk in the operational relay race.

Reduce access

They say too many cooks spoil the broth and that can apply to IT processes, as well. Limit the number of people who can directly make changes in production and you have a shot at increasing security and defending availability by reducing random acts of deployment, as well as change collisions and mystery changes that just show up in production. This is a key anchor for creating predictability and reducing change collisions.

Create defensible “choke points”

Reducing the number of ways things can be deployed into production will help you defend the integrity of the production environment. This doesn’t necessarily mean preventing changes, it is more about controlling how changes enter the environment. Focus on creating trustworthy repositories, deployment sources, automation tools, and service accounts that reduce the options for how deploys happen. When you define clear swim lanes and you know what the expected path of deployment looks like, it is easier to detect unexpected activities and actions.

Use choke points to enforce acceptance criteria to ensure that you are ready to take responsibility for the code, systems, and applications that being handed to you.

This approach allows great freedom in how work gets done in autonomous teams, while enforcing more consistency through the release management and transition to Ops phases of the life cycle.

Think like an air traffic controller

When dealing with multiple streams of work that need to come together and interoperate in production, I think a mental model of an air traffic controller works well. Airports allow a bunch of independent operators to come together in a shared environment in an orderly fashion, with minimal collisions and predictable throughput, and the air traffic controller plays a key role in making that happen. In the IT context, this means leaning on people who have a broader perspective than any individual team so they can look at the big picture and coordinate activities holistically.

This “zoomed out” perspective is critical in identifying risky activities and gaps, understanding dependencies that can break your security model, and stepping in to help mitigate dangerous conditions. This approach also makes it easier to work with third-party providers, since you can focus on their results (outputs and handoffs) rather than getting caught up in the minutiae of their day-to-day activities.

Monitor continuously to identify “trust gaps” and outliers

DevOps is no stranger to continuous monitoring and instrumentation. From a security perspective, continuous monitoring is about establishing a set of controls that allows you to quickly identify conditions that increase risk, and the elements above help you do that job more effectively. For example:
- Access limits and choke points enable you to quickly detect when someone has gone around your process and made changes directly in the production environment.
- These elements also allow you to keep tabs on third-party contractors and providers to ensure that they play by your rules, and you know what they are up to all the time.
- Defensible choke points enable you to sound the alarm if someone is tampering with your source repositories, automation scripts, or security monitoring components (such as disabling logging).
- Study your indicators to find dark spots in your telemetry around handoffs, and ask how an attacker (internally or externally) could exploit any weaknesses in your handoffs.
These are just a few examples of how these concepts can be applied. When you recognize that handoffs are risky, you can quickly see how security can play a role in mitigating those risks and helping defend the reliability, safety, and trust of your production environment.
May 12, 2014
Security automation with DevOps: show me the code!
Last week Andrew Storms put up a good post hinting at the promise of security automation in [SecDevOps: Security Automation By Example – The Firewall Change]. He included an example of automating a series of actions when a firewall rule is changed. It’s a good article, although I’m increasingly convinced there’s no such thing as SecDevOps. In my book, it’s all DevOps, but that’s fodder for another post (when I’m not battling a stomach bug). However, what Andrew describes is more of what I consider an automated assist. It isn’t necessarily full automation, since it triggers on a manual firewall rule change. Ideally we rarely manually change a firewall (or Security Group) rule, and rely more on self-configuring based on policies. Yeah, I know, the usual analyst BS, so here’s a bit of process, and a bit of code. Let’s approach this differently. Take Andrew’s process, but let’s have the security group or host firewall self configure based on the asset. My example is going to be Amazon Web Services specific, since I have some code snippets to show off how it works. (Sorry, I don’t have the time or intestinal fortitude to write out all the code today; seriously, where did this bug come from?!?) There are two techniques I generally see forming the core of dynamic Security Group policy enforcement:
1. Tags
2. Configuration Management integration
Some cloud services, like AWS, support tags at the object (in this case, instance) level. These tags can tie internally to policies while also being consumable for external tools. Here is a policy I use that restricts access to an object if the “SecurityStatus” tag is “IR” (for “Incident Response”):
```
{  
  "Version": "2012-10-17",  
  "Statement": [  
    {
      "Action": "*",
      "Condition": {  
        "StringEquals": {
          "ec2:ResourceTag/SecurityStatus": "IR"
          }
          },
          "Resource": [
          "*"
                      ],
          "Effect": "Deny"
      }
      ]  
    }         
```
I could apply that tag manually or automatically, and in either case the instance can no longer be managed by any account this policy is applied to. Now this doesn’t help us change firewall/Security Group rules, since you can’t manage those directly in EC2 based on tags (but I had the policy handy to demo, so there you go). For that to work, you need to write your own code to scan for tags and then take actions. I don’t have exactly the right code examples on hand. but here are some snippets to get you close. Most of these are cribbed from my SecuritySquirrel proof of concept on GitHub. They are all in Ruby, using the AWS 2.0 developer preview SDK. First, to pull a list of all instances with a certain tag, you can use the following:
```
def testing
 # testing some tag code
 instancelist = @@ec22.describe_tags(
 filters: [
 {
   name: "key",
   values: ["SecurityStatus"]
 }
 ]
 )
 puts instancelist.to_h
end  
```
That shows everything with the SecurityStatus tag (instances and values, shown as a hash), but you could also filter on value. From there, you could pull the instance IDs with a map (you might want to further filter only on instances, but if you made it this far you can figure that out):
```
 instances = instancelist.map(&:resource_id)  
```
Then you can pump those into a method to change the Security Group. Here’s the one from SecuritySquirrel that places an instance into a “Quarantine” group:
```
def quarantine
  #this method moves the provided instance into the Quarantine security group defined in the config file.
  puts ""
  puts "Quarantining #{@instance_id}..."
  quarantine = @@ec22.modify_instance_attribute(instance_id: "#{@instance_id}", groups: ["#{@QuarantineGroup}"])
  puts "#{@instance_id} moved to the Quarantine security group from your configuration settings."
end  
```
Want to trigger some scanning? Here’s a code snippet to trigger a Qualys scan on an instance from a virtual appliance in your AWS account (keep in mind, I’m skipping a bunch of important pieces, like configuring the service and authenticating):
```
  instance_IP = instance_details.reservations.first.instances.first.private_ip_address
 timestamp = Time.new
 scan =(HTTParty.post("https://qualysapi.qg2.apps.qualys.com/api/2.0/fo/scan/",
      :basic_auth => @qualysauth,
      :query => { 
        :action => "launch",
        :scan_title => "SecuritySquirrel Scan at #{timestamp}", 
        :ip => "#{instance_IP}",
        :option_title => "Initial Options",
        :iscanner_name =>  "us-west-2" 
      },
      :headers => { "X-Requested-With" => "ruby httparty"}))
      puts "Launching Qualys scan named: SecuritySquirrel Scan at #{timestamp}"  
```
Or want to change the CloudPassage Halo host firewall rules? Well, that code uses a pre-release of their Ruby SDK, so I’ll save it for another day (I’m not employed by them, but they let me play with the pre-release gem). These examples are all based on tags, but remember that if you use a tool like Chef or Puppet you can do the exact same thing based on what software is configured and running within the instance. For example, you could create a policy to quarantine all SSL when the Heartbleed vulnerability hit by closing off port 443 on any server running the vulnerable OpenSSL package, then remove the rule when it is updated to the patched version. I’m a bit rusty since I don’t code every day, but I could probably pop that code off in an hour or two, it isn’t that hard. Also my tool is designed for demos, and doesn’t run in the background, but clearly you could have it running all the time, adjusting things based on tags or even attributes of software deployed in the instance. Security automation is insanely powerful. It can completely automate complex security tasks, or, as in Andrew’s example, supplement manual activities and policy changes. Hopefully my snippets give you some ideas, and pop me over any good examples if you are willing to share.
May 7, 2014
SecDevOps: Security Automation By Example – The Firewall Change

Security Automation By Example

The Firewall Change

Just when you thought DevOps was the new black, along comes SecDevOps. Yes folks, like most things in life, the new cool is already here. Before I move on to trying to explain SecDevOps, please accept my mea culpa because for many people DevOps is yet to be clearly defined. I can imagine your frustration that I’m going to try and explain a new term based on a loosely defined term.

I’m not going to try and provide a complete definition of SecDevOps, but instead I’d like to provide what I believe to be important attributes of the SecDevOps movement. In this post, lets discuss security automation and use a firewall change process as an example.

One important attribute of SecDevOps is heavy use of security automation. In particular when I think security automation, I tell people to use IFTTT thinking. Stolen from the service whose tagline is “put the Internet to work for you”, I explain security automation simply as “if this, then that”. Its a highly simplistic, but poignant way to imagine ways in which one would automate workflow.

A simple act of changing a firewall has huge potential for security automation. The SecDevOps firewall change process goes beyond changing a few lines in a config, but also provides an independent and automated verification process that most organizations don’t perform today.

Consider what happens when a firewall policy is changed. First the policy change request is reviewed and approved. Then the policy change request is assigned and acted upon. Many times, but not always, the change is verified by another person through configuration review.

Security automation takes this process one step further. Imagine when a firewall is changed that a security audit event is automatically logged. Next, an automation component notices the firewall change and automatically spins up a remote scanning instance in a cloud provider. Next, that scanning instance is contacted using APIs to perform a remote network scan. The scan completes, results are exported and compared with prior scan results automatically. Finally, the automation component automatically updates the firewall change request in a ticketing system showing the scan results. We have now completed an automatic, independent and autonomous firewall change verification. That simplified yet powerful workflow is security automation and completes what many people refer to as a closed loop.

The new SecDevOps method for a firewall change not only provides a closed loop process, but does so by automating the verification steps. Joining traditional security processes with new API driven cloud services is a true force multiplier that many organizations will soon wish they had.

In upcoming posts, I’ll also discuss some of the other important components of what I believe make up the new black of SecDevOps.

April 30, 2014
You have to crawl before you walk…

In previous posts I have explained what Security Policy Orchestration is, why DevOps folks should care, and how it can help facilitate the cultural change necessary for organizations to reap the long-term benefits of the cloud and virtualization. In this post, I’ll provide a few examples of how Security Policy Orchestration can create some “quick wins” – wins that create common ground between Security, Dev and Ops, and set the stage for higher profile wins with significant business impact.

As infrastructure becomes increasingly complex it gets harder to decipher. In software we call this Readability – it determines our ability to support the code over time and to ensure security. The same concept applies to networks – the more complex they are, the harder it is to use them, maintain them and ensure security.

The primary value of Security Policy Orchestration is providing a holistic view of the infrastructure. It “sees” the entire network and creates an index, which enables network administrators to perform quick queries across their environment providing “needle-in-a-haystack” search capabilities. While it might seem obvious, gaining visibility into the environment is a very powerful tool that can deliver huge benefits, especially when the environment is complex and subject to constant changes.

Let me give you a few use cases:

Audits: As my security brethren know, preparing for an audit can take weeks (if not months).   However, once the network infrastructure is indexed, meaning you have structured records of the network elements (devices, interfaces, routes, security rules…), the kind of documentation required for security audits for PCI, SOX, NERC, and other regulations is literally a few mouse clicks away. Imagine finding all applications flows involved in a PCI flow in one click, or, even better, through a REST API call.

Many of Tufin’s large enterprise customers report that their internal audit requirements are more stringent and time consuming than external compliance mandates, and require customized reporting, which they often have to script themselves. Policy Orchestration solutions enable them to be much more agile – to the point where security no longer becomes a “showstopper.“

A successful and effective audit (one that actually improves security) requires a collaborative approach across Dev and infrastructure – having the needed data at hand ensures agility, which is crucial for embedding this into the continuous process.

Server Migration: Many enterprises and service providers are currently tasked with server migrations to virtual data centers or cloud environments. The challenge they face is identifying the talkers, connections and protocols that are needed to maintain connectivity and business continuity. Firewall policies are often the best place to find this information because their important role in “attaching applications to the network”. However, it’s a labor intense task. Even if you have only 10 firewalls, with a few hundred rules and a thousand objects, identifying the flows is going to take a while. Now multiply that by 100 migrations per month – it’s virtually impossible.

However, once you automate the process, it becomes quite easy. With change automation in place, all you have to do is click “Find”, or, even better, call an API, and you will know the flows immediately.

Application Deployment: As most devopsy.kinsta.cloud readers are (likely) well aware, when an Application team needs a connection opened to say, an LDAP database, they open a ticket with a business request: “I need to connect server A to LDAP database B.”

It’s the network guy (or gals) job to translate that request into a set of IP addresses that plot a secure, efficient network path that connects the database to the application. A week or so later, the requisite changes are made, and voilà, the application can leverage the LDAP database. Or not, and then the network team needs to re-trace its steps and figure out what the problem is.

The above process, in many organizations, is a loose one, usually conducted via email, and between people who don’t often traverse the same orbit.

Add human error into the equation – a mistyped or incorrectly documented IP address, and you end up with re-dos, which everyone hates because they are time consuming and in complex network environments can be impossible to trouble shoot manually.

Ask any network manager – reducing (if not eliminating) re-do’s is an impressive feat that offers substantial efficiency gains.   It’s also worth noting that without automation it is really, really hard to properly risk assess proposed changes. In worst-case scenarios, an insecure network connection becomes apparent after something bad has happened. But why wait for something bad to happen before innovating? Bring security into your next network virtualization initiative at its onset – I promise you, you won’t be sorry.

DevOps folks are likely familiar with the idea of abstraction, but it is still a relatively new concept in network security circles.   When applied to network change management, Abstraction separates business logic from the technology underpinnings of network infrastructure, and allows Operations teams to translate higher-level application language into terms they can use to enable underlying network services.   In scenarios like the above, the ability to automate this process can not only deliver huge efficiencies and create a much more collaborative dynamic between application, network and security teams.

Hopefully, these examples build the case for how Security Policy Orchestration can be leveraged to deliver some quick wins. Of course, organizations that commit to deeper and wider change reap greater benefits. However, as my Co-founder, Tufin CEO Ruvi Kitov likes to say, you have to crawl before you walk, and walk before you run. So in my next post, I’m going to explore why security people are so weary of automation, and how to get around that, in hopes that Security, Dev and Ops teams can all walk together.

April 29, 2014
DevOps: Security’s last best hope

Help us Obi Wan! The fact is that DevOps is security’s last best hope. The sooner the security industry realizes it the better it will be for everyone.

I read George Hulme’s story today about proceeding with caution and going slow with DevOps. Of course it was no surprise to me that those cautioning to go slow were my fellow security industry professionals. What else do you expect, after all we are the “people who say no”. At least the three folks George interviewed didn’t flat out say no to DevOps. They just advised caution. The fact is that DevOps is security’s last best hope. The sooner the security industry realizes it, the better it will be for everyone.

Let’s be honest. The security industry for the most part has not been a stunning success in keeping our data, apps and infrastructure safe and secure. Some may even say it has been an abject failure. What do you expect though? The game is stacked against us. Almost by definition the entire security industry is by definition a reactive model. We are always one step behind, reacting to the newest moves of the bad guys. To be a security professional is to know failure.

What is worse, we are generally ignored until we fail and then the harsh, bright light of a security breach shines brightly in our eyes. Is it any wonder that so many security folks feel that they are not successful? Our budgets are never enough, we never have enough resources, no one ever does what we ask and not enough people take security seriously enough until after the doo doo hits the fan. Yes it takes a special kind of masochist to be a security person.

It is not by happenstance that in Gene Kim’s book, The Phoenix Project, the poor bastard of the security guy goes out on a bender and comes back at the end having fallen off the wagon?

For all of the aggravation of fighting an impossible to win fight how are security people viewed by the rest of the IT folks and others in the organization. I am sorry to say that all too often we are though of as the crazy person in the back of the room yelling no. Again as in Gene’s book, it gets so bad that people stop inviting us to their meetings. They know what we are going to say, NO, so what’s the point.

There are more enlightened or maybe smarter security folks who don’t say no any longer. Now they say yes do it, but let’s go slow and make sure we understand it. When asked why, these security folks have a built in scape goat. They blame compliance. Yes, f**king compliance. Yeah, that’s the ticket. We have to go slow because our compliance obligations require it. I laughed out loud when I saw this raggedy old flag being hoisted in George’s article.

Compliance is always the fallback of the security guys who say no. I have seen many examples of this over the years. Two of my favorites are:

1. The security guy who asks the IT manager or C-level guy “how do you look in stripes”, because if you do this without going slow and making sure we are compliant, we could all wind up in the slammer.

Hell we didn’t send anyone to jail for bringing this country to the brink of economic ruin. Do you really think we are going to send someone to jail because by adopting DevOps we might blow a PCI requirement? Can we really do any worse than Target?

2. Walking around a US Army base talking to the security officer I asked him how he was dealing with the compliance issues around securing wireless networks on base. He replied that was easy, he forbid wireless networks on base, so they were in full compliance. Of course as he said this I was watching people unplug their WAPs and throw them under their desks as we walked by.

Yes, security folk can fall back on compliance as the reason for our no, but it is tired and old. The same for the security person who are always promising the next magic bullet. If only they had the latest shiny, new budget busting trinket they could finally sleep well at night. Even if they get their new toy, it only turns up that the bad guys have found a new vector and we need the next silver bullet.

No, I am afraid being the guys who says no or even the people who say OK, but let’s go slow is just not going to cut it. The speed of business today at the enterprise or the startup is anathema to the “go slow” mantra.

On the other hand, DevOps offers a new way. A way to change the way the game is played. For so long I have heard security folk wax poetic about the possibility that security could be built in to the process. Instead of being bolted on after the fact, wouldn’t it be great if it were built in by design? Wow, what a great thing that could be.

DevOps has the potential to deliver on the built in dream. DevOps can make the developer more aware of security and even compliance issues while the code is being written. It can give the security team insight and input into the development planning and process.

DevOps has the potential to break down the silo that separates security from the rest of IT. Just as it brings Dev and Ops together, DevOps can give security a seat at the table. Security can move from being the crazy madman in the back to a respectable member of the team.

This security transformation won’t be easy though. It will demand a change in security’s attitude. First and foremost, we have to change from being the people who say no or go slow, to the people who say “yes we can”. We need to understand that many of these projects and initiatives are going to move forward with or without us. Better to get on the bus than to get run over by the bus.

Next we have to do a better job of communicating our concerns or issues. Falling back on FUD or compliance is not cutting it. We need to communicate the risks involved, how we can mitigate the risk and understand that a business decision will be made as to what level of risk is acceptable and how that risk is going to be managed. At that point we have given enough information for the business people to make a business decision. They make the decision, we get behind it, we don’t bitch bite and we move forward.

Security can’t be the people who say no or go slow. When security starts to say yes we can and this is how we do it, security will be given the seat at the IT table that we deserve.

April 24, 2014