devopsy.kinsta.cloud

Tag: security

Ensuring security and managing risk in enterprise DevOps

Today’s question in this recurring series comes from a developer in a typical large enterprise – risk averse, tightly controlled, highly secured – a major challenge to any DevOps transformation:

Q. I work in a large government department that is tight on security. We have strict policies on separation of duties, can’t share users IDs or passwords, struggle to share code, can’t use AWS or Azure, audit everything, and so on. How can DevOps work in my environment?

If DevOps is about collaboration and removing bottlenecks, then it must include security, from planning to coding to testing to operations; building, delivering, and running applications that are secure by design. Unfortunately, for many enterprises security is a major roadblock in the application delivery lifecycle.

Yet security does not need to be a DevOps bottleneck. Indeed, these specific issues are solved problems. While I do not boast of a deep security expertise, I can address the solutions based on my experience (and some further reading – see below):

Maintain separation of duties

Large enterprises must maintain a separation of duties, to prevent (for example) a rogue developer from adding a back door to new code and putting it into production themselves, with no checks and balances. But staff can still have the access they need when they need it – and no more. When roles change, such as developers accessing the operations ‘flight deck,’ or operators being seconded to development, identity management and access controls can provide the right access automatically, as needed, audited and policy-compliant.

Protect privileged user IDs and passwords

For large enterprises it is never acceptable to share passwords, especially to privileged accounts like cloud control panel admins. However, privileged identity management allows staff to ‘check-out’ privileged IDs from a secure pool, use them as needed (e.g. for break-fix), then check them back in when they are done. As policy dictates, access can be subject to additional approvals, time limits, access limits, keystroke logging, session capture, etc. This allows on-demand access to sensitive IDs while maintaining stringent audit and control.

Provide secure cross-team interfaces

Part of DevOps is sharing – processes, skills, knowledge … but also reusing code and services (or microservices) through shared APIs, whether public or proprietary. This sharing accelerates delivery and reduces costs, but can also introduce risk. To prevent unintended consequences when sharing code and services, API security can enable sharing both internally and externally, but with controls like user authentication (e.g. OAuth), granular access controls, volume/rate limiting, blocking and more.

Enable secure on-demand cloud access

Most enterprise staff cannot just fire up an AWS or Azure instance on the corporate card. Even applying for ‘approved’ access can be a multi-week process. Instead, security pros can set up automated self-service provisioning to ‘known good’ or ‘enterprise-grade’ services, pre-approved by security, HR, finance, etc. This helps to maintain necessary audit and control while enabling DevOps staff to quickly and easily adopt cloud and other services.

Use automation to maintain security and governance

Automation is a powerful way to reduce risk, ensure compliance, and maintain governance. For example, release automation allows developers to deploy new code without having personal access to production systems (the tool gets the access, not the individual). It can also restrict access to such systems through ‘known-good processes,’ to ensure deployment complies with policy. Tracking who initiates deployment maintains that essential audit trail for governance and reporting. Test automation, data management, process automation, configuration management and other automation tools all provide similar security benefits throughout the application delivery lifecycle.

Have a security culture

Of course, there is more to secure DevOps than technology solutions. Just as DevOps requires a new culture of collaboration, good governance ultimately requires a culture of security. Security is everyone’s responsibility at every step of the service delivery lifecycle. Security is still a specialist role, so security pros should have a place in agile planning, standup and triage meetings, but good governance is everyone’s role. Dev, ops, test, QA and others all need to step to the security challenge; and security pros should share their requirements, knowledge, best practices and tools with the rest of the DevOps team.

Further Reading

In compiling this post, I leaned on many others who have helped to establish the concepts of DevOpsSec / SecDevOps and Rugged DevOps. A great starting point for more specialist detail on this topic is the SecDevOps content here on devopsy.kinsta.cloud, particularly the work of Rich Mogull (@rmogull) and Andrew Storms (@st0rmz). For an early intro, check the Gartner note, DevOpsSec: Creating the Agile Triangle and the seminal work of James Wickett (@wickett) on Rugged DevOps. For a more recent thought, try the Threatstack blog, and Edward Haletky (@texiwill) for a good end-to-end view. Finally, for a great resource specific to DevOps auditability, check the Gene Kim (@RealGeneKim) project, the DevOps Audit Defense Toolkit.

September 22, 2014
Internet of Things (IoT) poses challenges for DevOps and security

One of the hottest things in tech today is the Internet of Things (IoT). Everything will be connected including every gadget, kitchen appliance, wearable device and vehicle. But one of the biggest areas for IoT are controllers embedded into just about everything. These controllers will report back, generating reports and information that can be analyzed and acted upon. IDC says technology and services revenues rooted in components, processes, and IT support for the IOT will reach US$7.3 trillion by 2017, up from US$4.8 trillion in 2012. When you are throwing around numbers in the trillions that is no small feat.

New IoT implementations require projects (such as application development and backend integration, for example) that many enterprise customers don’t have the resources or custom development experience for. Vendors such as Microsoft and AT&T are responding with IoT consulting services. These vendors must be flexible to meet the needs of enterprise customers whose IoT can include machines and devices from a variety of makers. As such the bigger vendors are generally agnostic towards partnering with otherwise competing vendors in order to pool the necessary resources to complete an IoT project.

When management contracts a vendor / consultant for this kind of engagement, a DevOps mindset really helps to ensure the new IoT ‘glove’ fits the existing IT and information security ‘hand’. As challenges appear, a DevOps approach utilizing best practices is the best way to meet them.

A common challenge with IoT projects is the disconnect between management desires and IT realities. IoT projects often begin because management has a goal to receive specific information. Many IoT projects involve installing controllers in business and manufacturing to retrieve data from their IoT. The controllers themselves are of course hardware. This of course is a challenge to traditional DevOps approaches. By its very nature IoT is dealing with hardware that is difficult to upgrade. Of course more and more it is the software that is driving the innovation

“In many cases, management may hear good things from a competitive company they know who uses a specific controller device that they really like,” says Jerry Irvine, CIO, Prescient Solutions. Management may request that the vendor pull in that maker and product and that IT work with them to make it work. Unfortunately, IT will often find that the device is not compatible with the proprietary systems they use. Then the business will often ask them to find a compatible substitute and may still want it by a given deadline. This is not how anyone should acquire controllers.

The best practice solution is for the enterprise to have an R&D team including IT set up to work in an on-going fashion to provide requirements for the IT environment and the business function. “In this way, IT and executive management can proactively and consistently look for new offerings with compatible features and functionality,” says Irvine.

Another way DevOps can help in IoT projects is to maintain the environment in the least customized way possible. When a vendor puts in a controller, they will certainly be able to customize it to provide the specific reports and functionality that the customer wants. If the customer wants to go beyond the standard reports and capabilities however it now can pose a challenge to automation and further development.

The issue is that when the enterprise needs to apply updates to the hardware, device software, or management application, the vendor must customize that as well, and the updates come at a hefty premium rather than standard (gratis) with the product. The best practice solution, says Irvine, is for the enterprise to take the entire report that the controller can provide standard, and customize the data from the report on the end afterwards. Customize as little as possible.

Security is another challenge for IoT projects. The fact that these devices are connected via the Internet can make them a target for hackers. Companies deploying controllers should ask themsevles “how sensitive is the data that IoT devices and wireless communications will handle, and how dangerous can the IoT machine or device be? ” If developers/IT and operations can say with a certainty that the IoT application will never handle extremely sensitive data such as credit card numbers, and that the IoT device is not a car, dangerous machinery, or a potentially dangerous medical device, for example, then of course security is less of a concern. But part of goal of DevOps should be how do we build security organically into the device, software and process.

John Pescatore, Director of Emerging Security Trends, the SANS Institute. IT and management says we will have to limit what IoT applications can do. If the application must use sensitive data, it has to be encrypted and the end device has to have enough processing power to do encryption or it cannot be part of this application. Likewise, for security of life and death critical applications, there is a part of the DevOps model that does not make sense. That part is the “build a little/fail a little/fix” type model of the evolution and perfection of the device. “Letting your customers find failures in MRI machines or 787 Dreamliners is not a good idea,” says Pescatore.

So how do we reconcile the DevOps model, along with security when building components for the IoT? The challenges are there, but they can be overcome when we are talking about trillions of dollars at stake.

August 25, 2014
Building Security and Compliance into the SDLC

As the concept of DevOps has become more popular in recent years, there are some common themes that always seem to come up in discussions about it. For example, how often do we talk about ways to speed up the software development life cycle (SDLC) in order to bring innovative applications to market faster and meet ever-increasing customer demands?

As important as this and other popular DevOps themes may be, though, they often have the side effect of overshadowing some of the less glamorous topics, such as security and compliance, which is often viewed as a sort of “necessary evil.” That said, the risks of putting an unsecure application into production can create massive challenges for organizations – from breaches to loss of company IP to brand damage – making security and compliance a topic that must be tackled as enthusiastically as accelerating innovation.

Three Critical Areas of Focus

When it comes to considering how security and compliance can become more integrated with DevOps practices, there are three critical areas on which all organizations should focus: shifting security left in the SDLC, building security into policies and creating an audit trail throughout development.

Shifting Security Left in the SDLC

Organizations that incorporate security in the earliest stages of the SDLC will ultimately benefit from products and applications that are secure by design. But in order to accomplish this, they can’t wait until the software hits production to test it for vulnerabilities.

Service virtualization can help by simulating the application delivery run-time environment and its data, allowing developers and testers to focus on security immediately, instead of waiting for access to crucial resources. In addition, a release automation tool can help collapse development cycle times from days to minutes, enabling IT teams to spend more time improving the functionality, quality and security of their applications – and less manually promoting them from stage to stage.

Building Security into Policies

It used to be that every element of application development took place between an organization’s four walls, but with the rise of cloud and software-as-a-service (SaaS) technologies, it’s not uncommon for different stages of the SDLC to occur on different infrastructures. So if the environments are always changing, how do we ensure security is consistently addressed? By building it into the policies that comprise our continuous validation methodologies.

Release automation enables development teams to create policy-driven infrastructure provisioning rules that automatically determine what application elements can and cannot be staged on specific infrastructure types. For example, a common policy would be to never let customer or database information move to a public cloud environment, where sensitive information could be lost or stolen.

Creating an Audit Trail through Development

To this point we’ve centered the conversation mostly on security, but being able to maintain – and prove – compliance throughout the SDLC is an equally important challenge that IT development and operations teams must solve.

So when we think about compliance, we’re really asking questions like “What application moved from this stage to that stage?”, “Who released the application into production?” and “If there was a rollback, what was the cause and who rolled it back?” In other words, we’re talking about audit trails and logging all activities within the SDLC – something that can be difficult when numerous manual steps are involved. However, a release automation tool will capture all of this data as a matter of course as it automates release deployments through orchestration and promotion of applications from development through production. This means you’ll always be able to answer those who, what, where and when questions that pop up during audits.

Eliminating Vulnerabilities and Getting Back to the “Good Stuff”

I’ll admit that security and compliance are not the most exciting topics in DevOps. If application development were a meal, they’d be the pile of lima beans you have to eat before you get to the dessert that is innovative feature development. The good news is, service virtualization and release automation tools can help you speed and simplify the process of building security and compliance into the SDLC, which ensures your applications will hit the market without vulnerabilities – and lets you get back to the fun stuff.

July 7, 2014
Security in a DevOps world: Q&A with Gartner’s Ben Tomhave

As the infosec community catches up with the spirit of the DevOps movement, many CISOs and IT security leaders still need guidance on how to make the paradigm shift. Not only are these security folk scratching their heads to figure out the additional risks posed by continuous delivery, but they’re also learning how DevOps can actually help them improve their effectiveness and within their IT ecosystem.

Gartner recently took a deep dive into this topic with a report entitled “Security In a DevOps World”. devopsy.kinsta.cloud caught up with one of the lead authors of the report, Ben Tomhave, research director for Gartner, to discuss some of his findings and to learn more about how he thinks security staff need to adjust philosophies and practices to peacefully co-exist with DevOps.

devopsy.kinsta.cloud: How do you think security people need to change their philosophy in order to get the most value out of DevOps practices?

Tomhave: Getting away from that adversarial ‘party-of-no’ attitude is very key. We have to get away from “Come to us, we’ll give you things” and start thinking of partnering, collaborating and mentoring. On the development side, security needs to be pulled in earlier in the process, providing faster feedback after code commits. And then it needs to be working with operations to provide secure foundations, really inherently secure environments that are secure by default so that nothing is changing after the code is written.

Because that’s been a huge problem, too, right? Where developers go write code in who knows what environment and then it propagates to QA, which looks a little closer and sends it to production. Then it launches and suddenly there are all these surprises, things are breaking left and right and there was no communication along the way. Everybody was working on different platforms, and ops maybe wasn’t even managing or maintaining the development platforms. We’ve got to get away from that and security is no different—it should be a part of the process. All the monitoring tools, all the hardening tools, all that stuff needs to be applied from dev through QA to production.

devopsy.kinsta.cloud: What would you say are the biggest security concerns or risks that crop up in DevOps environments? What kinds of things could go wrong if security teams aren’t embedded in the DevOps patterns?

Tomhave: I think that it ‘s the same old stuff, pushing out code that may be functionally correct but woefully insecure and all of the other gotchas, whether it be bad crypto, bad handling of identity management and insecure management of data. DevOps shops are not immune to these problems, either. One need only look at the Spotify disclosure that they had password resets because of a breach. That’s a big DevOps shop and clearly some error was made.

The good news is that a DevOps shop that’s doing lots of code pushes should be able to resolve issues relatively quickly. But that means it is incredibly imperative that they work closely together so that when there’s a break it is quickly triaged, remediated rapidly and there’s good clear communication between the incident responders–whether they be security incident responders or operations incident responders—and developers, getting the feedback back to developers in a timely fashion.

devopsy.kinsta.cloud: It’s clear from your report that you think monitoring is more important than ever to support the rapidity of the DevOps cycle. Why is that?

Tomhave: You know, we expanded that DevOps mantra of fail fast, learn fast to add recover fast in there, too. If you want to fail fast, that’s cool. And if you want to learn fast, that’s also cool. But you also need to have rapid recovery as well, because the sooner you can detect a breach or an incident of some sort or a security event, the sooner you can start interdicting and remediating and taking positive actions to clean it up.

If you do DevOps with lots of operational transparency into process, but you don’t have adequate transparency into what the project is doing with the data involved and overall security monitoring capabilities, you could have a lot of problems. Suddenly now we’re going to be rapidly iterating through features and code pushes that are expanding capabilities and then we never know whether or not that environment is being attacked or whether or not it is being adequately secured or if it is breaking down. So that transparency meme needs to expand out to security monitoring as well so that you have security operational transparency as well as process transparency.

devopsy.kinsta.cloud: I was really interested in your advocacy for a security architect to help facilitate security meshing into DevOps. What does an organization need from this role and what are the challenges of finding the right person to do the job?

Tomhave: The challenge here is what you really need is someone who is really competent with development and preferably has some actual experience that allows them to establish credibility with the development team.

But they also must be really strong security people as well. So they understand static analysis, dynamic analysis, they understand what he tools are doing, what things are looking for. They must also understand design principles from a security perspective, things like here’s why you don’t roll your own crypto, why you integrate with existing federated identity systems, why you do transport encryption and what an appropriate level of protection is for a given system.

Finding those two sets of qualities in a single person can be really challenging. It is really difficult to find them and, more importantly, once you do find that person, they tend to be very expensive, especially for smaller shops. You think of all these mobile start-ups, for example. Their feet are being held to the fire; they’re being subjected to regulator actions. We’ve seen the FTC go after some of these mobile app companies for violating privacy policies and things like that. So how do you get those security bodies in there when you may not be able to afford to hire someone?

Alternative staffing models really speak to that. They may need to find consultants through specialty shops or something along those lines to get someone to come in and be engaged either through a staff augmentation type of role or just on a per-project basis.

I think the big key is being able to do that around the design review up front and catch some of these problems early. Because no matter how much you invest in tools and how many times you scan the code base–either statically, dynamically or whatever–that’s only going to check maybe 40 percent of your code. And a lot of these functional weaknesses, you aren’t going to be able to detect them with tools. You have to have somebody look at the design and say, ‘Oh, you’ve forgotten completely to do encryption,’ or, ‘Oh, I see you’re hashing, but you haven’t salted your hash, so how are your passwords secure?’ or ‘Why are you dealing with credit card data here, this shouldn’t be done this way, you should just be interconnecting into our payment system over there. Quit reinventing the wheel.’

Those types of things need to be caught by architects early on, rather than trying to catch it using tools in the code.

devopsy.kinsta.cloud: Speaking of tools, you dedicated time in your report discussing the importance of tool chains and automation. Automation has always been important to security, but what’s changed now with DevOps?

Tomhave: Well, I think that the big thing is almost all of the testing has to be automated up front. Static code analysis should just be automatically triggered on a nightly repository of check-ins, as much as possible. Same with dynamic analysis. Dynamic app sec testing should be done automated and feedback should just be fed back directly into the issue management systems. There shouldn’t be much human touch there at all, aside from monitoring for, addressing and resolving false positives in those tools. And possibly adding some customized checks looking for specific libraries or frameworks that should be in place or should be used within the tools.

The only time where I think we want to have any sort of manual interruptions are the first up-front quick risk analysis. Deciding is this a high risk project that’s dealing with sensitive data or has uniquely high uptime requirements, or is this a lower risk project where we’re not really concerned about losses here? I think that, as well as some of the design review considerations would need some manual hands and eyes involved.

And then everything should automate until you get to the end where you say, ‘OK, this is a high risk project, we need to put extra testing into it, so now we’re going to kick off pen testing, where we need to have some manual validation in place.’ That validation not only deals with false positives but really does a deeper dive into the application. So you fuzz the interface, it looks like that broke some stuff and you say ‘Yeah, we validated and proved it and here’s the exploit. This is how it broke.’ Then feed that back in a credible fashion to developers to show there’s the spot in the code with the errors or this is the logic error that led to whatever the condition may be.

June 2, 2014
Deputizing Everyone for Security – Building Agile Assurance

Those of us highly focused on the delivery pipeline of DevOps will wonder why we should include the security guy to the party. After all, aren’t they just going to slow down the process and make it harder to deliver good features to end customers?

My prior post about introducing SecDevOps by example was not about defining a new role, but introducing how security can and should be part of the automation delivery suite. Whether you are talking about traditional infrastructure change process or automated instantiation of new nodes to handle customer demand, the tools and processes of DevOps merged with security requirements are available to provide security accountability and assurance. Those of you coming from the traditional security space, which might fear security automation, should consider just how exciting SecDevOps delivers on agile assurance.

Rich Mogull provided an excellent example and code to show how firewall changes in a real life situation can be fully automated. There is no need to be squirrely about what is emerging. Everyone needs to be involved with development and operations to take more opportunities to minimize risk.

Dwaye Melancon also recently wrote about the issues of security and risk when the baton is handed off from one owner to another. This too is another great example of where process and monitoring automation aids in delivering continuous security assurance.

I believe everyone craves for a future when security events are automated with full transparency from beginning to end. If security had a real closed-loop transparent agile process, then just imagine how easy it could be to deliver on both compliance and real risk reduction. As Richard Stiennon recently stated in a CloudPassage webcast “Compliance standards are woefully behind with the reality of operations”. Haven’t we all had an experience with an auditor and a clipboard asking for specifics that are clearly aligned with another age of computing?

Wouldn’t it be better to give auditors the same tools, to get a picture of accountability and risk decisions made in the DevOps cloud environments? Take for example the recent HeartBleed events that many security teams are still struggling with. As a DevOps professional, I would have rather just showed the security team and the auditors the truth – that none of my team did anything at all to mitigate this risk. Truth be told, our orchestration tools had already installed the new version of OpenSSL and generated audit logs before many people had their first cup of coffee.

Security can be improved with more integrated automation and by deputizing everyone to do more with DevOps tools. A SecDevOps approach will improve the compliance and audit process with tactical security tools for all. Burn the clipboards and provide an enlightened way to prove best security practices from the collective group of Dev and Ops. Maybe give the Security guy a break and invite him to the after audit beer bash too.

May 21, 2014
Security in the operational relay race
If you’ve ever seen people run a relay race, you’ll notice that most of the risk happens when the baton is handed from one runner to the next. In IT processes, there is also dramatically increased risk during handoffs. Whether you’ve moving from one process phase to another, one “owner” to another, or one environment to another the risk is greatest during the phase transition.

Security can play a strong role during these handoffs, whether you’re using DevOps or not. Here are some principles that can help safeguard the handoffs and reduce the risk in the operational relay race.

Reduce access

They say too many cooks spoil the broth and that can apply to IT processes, as well. Limit the number of people who can directly make changes in production and you have a shot at increasing security and defending availability by reducing random acts of deployment, as well as change collisions and mystery changes that just show up in production. This is a key anchor for creating predictability and reducing change collisions.

Create defensible “choke points”

Reducing the number of ways things can be deployed into production will help you defend the integrity of the production environment. This doesn’t necessarily mean preventing changes, it is more about controlling how changes enter the environment. Focus on creating trustworthy repositories, deployment sources, automation tools, and service accounts that reduce the options for how deploys happen. When you define clear swim lanes and you know what the expected path of deployment looks like, it is easier to detect unexpected activities and actions.

Use choke points to enforce acceptance criteria to ensure that you are ready to take responsibility for the code, systems, and applications that being handed to you.

This approach allows great freedom in how work gets done in autonomous teams, while enforcing more consistency through the release management and transition to Ops phases of the life cycle.

Think like an air traffic controller

When dealing with multiple streams of work that need to come together and interoperate in production, I think a mental model of an air traffic controller works well. Airports allow a bunch of independent operators to come together in a shared environment in an orderly fashion, with minimal collisions and predictable throughput, and the air traffic controller plays a key role in making that happen. In the IT context, this means leaning on people who have a broader perspective than any individual team so they can look at the big picture and coordinate activities holistically.

This “zoomed out” perspective is critical in identifying risky activities and gaps, understanding dependencies that can break your security model, and stepping in to help mitigate dangerous conditions. This approach also makes it easier to work with third-party providers, since you can focus on their results (outputs and handoffs) rather than getting caught up in the minutiae of their day-to-day activities.

Monitor continuously to identify “trust gaps” and outliers

DevOps is no stranger to continuous monitoring and instrumentation. From a security perspective, continuous monitoring is about establishing a set of controls that allows you to quickly identify conditions that increase risk, and the elements above help you do that job more effectively. For example:
- Access limits and choke points enable you to quickly detect when someone has gone around your process and made changes directly in the production environment.
- These elements also allow you to keep tabs on third-party contractors and providers to ensure that they play by your rules, and you know what they are up to all the time.
- Defensible choke points enable you to sound the alarm if someone is tampering with your source repositories, automation scripts, or security monitoring components (such as disabling logging).
- Study your indicators to find dark spots in your telemetry around handoffs, and ask how an attacker (internally or externally) could exploit any weaknesses in your handoffs.
These are just a few examples of how these concepts can be applied. When you recognize that handoffs are risky, you can quickly see how security can play a role in mitigating those risks and helping defend the reliability, safety, and trust of your production environment.
May 12, 2014
Security automation with DevOps: show me the code!
Last week Andrew Storms put up a good post hinting at the promise of security automation in [SecDevOps: Security Automation By Example – The Firewall Change]. He included an example of automating a series of actions when a firewall rule is changed. It’s a good article, although I’m increasingly convinced there’s no such thing as SecDevOps. In my book, it’s all DevOps, but that’s fodder for another post (when I’m not battling a stomach bug). However, what Andrew describes is more of what I consider an automated assist. It isn’t necessarily full automation, since it triggers on a manual firewall rule change. Ideally we rarely manually change a firewall (or Security Group) rule, and rely more on self-configuring based on policies. Yeah, I know, the usual analyst BS, so here’s a bit of process, and a bit of code. Let’s approach this differently. Take Andrew’s process, but let’s have the security group or host firewall self configure based on the asset. My example is going to be Amazon Web Services specific, since I have some code snippets to show off how it works. (Sorry, I don’t have the time or intestinal fortitude to write out all the code today; seriously, where did this bug come from?!?) There are two techniques I generally see forming the core of dynamic Security Group policy enforcement:
1. Tags
2. Configuration Management integration
Some cloud services, like AWS, support tags at the object (in this case, instance) level. These tags can tie internally to policies while also being consumable for external tools. Here is a policy I use that restricts access to an object if the “SecurityStatus” tag is “IR” (for “Incident Response”):
```
{  
  "Version": "2012-10-17",  
  "Statement": [  
    {
      "Action": "*",
      "Condition": {  
        "StringEquals": {
          "ec2:ResourceTag/SecurityStatus": "IR"
          }
          },
          "Resource": [
          "*"
                      ],
          "Effect": "Deny"
      }
      ]  
    }         
```
I could apply that tag manually or automatically, and in either case the instance can no longer be managed by any account this policy is applied to. Now this doesn’t help us change firewall/Security Group rules, since you can’t manage those directly in EC2 based on tags (but I had the policy handy to demo, so there you go). For that to work, you need to write your own code to scan for tags and then take actions. I don’t have exactly the right code examples on hand. but here are some snippets to get you close. Most of these are cribbed from my SecuritySquirrel proof of concept on GitHub. They are all in Ruby, using the AWS 2.0 developer preview SDK. First, to pull a list of all instances with a certain tag, you can use the following:
```
def testing
 # testing some tag code
 instancelist = @@ec22.describe_tags(
 filters: [
 {
   name: "key",
   values: ["SecurityStatus"]
 }
 ]
 )
 puts instancelist.to_h
end  
```
That shows everything with the SecurityStatus tag (instances and values, shown as a hash), but you could also filter on value. From there, you could pull the instance IDs with a map (you might want to further filter only on instances, but if you made it this far you can figure that out):
```
 instances = instancelist.map(&:resource_id)  
```
Then you can pump those into a method to change the Security Group. Here’s the one from SecuritySquirrel that places an instance into a “Quarantine” group:
```
def quarantine
  #this method moves the provided instance into the Quarantine security group defined in the config file.
  puts ""
  puts "Quarantining #{@instance_id}..."
  quarantine = @@ec22.modify_instance_attribute(instance_id: "#{@instance_id}", groups: ["#{@QuarantineGroup}"])
  puts "#{@instance_id} moved to the Quarantine security group from your configuration settings."
end  
```
Want to trigger some scanning? Here’s a code snippet to trigger a Qualys scan on an instance from a virtual appliance in your AWS account (keep in mind, I’m skipping a bunch of important pieces, like configuring the service and authenticating):
```
  instance_IP = instance_details.reservations.first.instances.first.private_ip_address
 timestamp = Time.new
 scan =(HTTParty.post("https://qualysapi.qg2.apps.qualys.com/api/2.0/fo/scan/",
      :basic_auth => @qualysauth,
      :query => { 
        :action => "launch",
        :scan_title => "SecuritySquirrel Scan at #{timestamp}", 
        :ip => "#{instance_IP}",
        :option_title => "Initial Options",
        :iscanner_name =>  "us-west-2" 
      },
      :headers => { "X-Requested-With" => "ruby httparty"}))
      puts "Launching Qualys scan named: SecuritySquirrel Scan at #{timestamp}"  
```
Or want to change the CloudPassage Halo host firewall rules? Well, that code uses a pre-release of their Ruby SDK, so I’ll save it for another day (I’m not employed by them, but they let me play with the pre-release gem). These examples are all based on tags, but remember that if you use a tool like Chef or Puppet you can do the exact same thing based on what software is configured and running within the instance. For example, you could create a policy to quarantine all SSL when the Heartbleed vulnerability hit by closing off port 443 on any server running the vulnerable OpenSSL package, then remove the rule when it is updated to the patched version. I’m a bit rusty since I don’t code every day, but I could probably pop that code off in an hour or two, it isn’t that hard. Also my tool is designed for demos, and doesn’t run in the background, but clearly you could have it running all the time, adjusting things based on tags or even attributes of software deployed in the instance. Security automation is insanely powerful. It can completely automate complex security tasks, or, as in Andrew’s example, supplement manual activities and policy changes. Hopefully my snippets give you some ideas, and pop me over any good examples if you are willing to share.
May 7, 2014
Security and DevOps showdown

George Hulme’s article on proceeding with caution on DevOps implementation offers some advice from a few well known security experts. Article here

vs.

Alan Shimel says today’s speed of business isn’t compatible with :go slow”. Security people have to understand that DevOps is our last best hope. But in order to have DevOps help us to security nirvana we need to change how we interact with the rest of our IT teams. Article here

April 24, 2014
DevOps: Security’s last best hope

Help us Obi Wan! The fact is that DevOps is security’s last best hope. The sooner the security industry realizes it the better it will be for everyone.

I read George Hulme’s story today about proceeding with caution and going slow with DevOps. Of course it was no surprise to me that those cautioning to go slow were my fellow security industry professionals. What else do you expect, after all we are the “people who say no”. At least the three folks George interviewed didn’t flat out say no to DevOps. They just advised caution. The fact is that DevOps is security’s last best hope. The sooner the security industry realizes it, the better it will be for everyone.

Let’s be honest. The security industry for the most part has not been a stunning success in keeping our data, apps and infrastructure safe and secure. Some may even say it has been an abject failure. What do you expect though? The game is stacked against us. Almost by definition the entire security industry is by definition a reactive model. We are always one step behind, reacting to the newest moves of the bad guys. To be a security professional is to know failure.

What is worse, we are generally ignored until we fail and then the harsh, bright light of a security breach shines brightly in our eyes. Is it any wonder that so many security folks feel that they are not successful? Our budgets are never enough, we never have enough resources, no one ever does what we ask and not enough people take security seriously enough until after the doo doo hits the fan. Yes it takes a special kind of masochist to be a security person.

It is not by happenstance that in Gene Kim’s book, The Phoenix Project, the poor bastard of the security guy goes out on a bender and comes back at the end having fallen off the wagon?

For all of the aggravation of fighting an impossible to win fight how are security people viewed by the rest of the IT folks and others in the organization. I am sorry to say that all too often we are though of as the crazy person in the back of the room yelling no. Again as in Gene’s book, it gets so bad that people stop inviting us to their meetings. They know what we are going to say, NO, so what’s the point.

There are more enlightened or maybe smarter security folks who don’t say no any longer. Now they say yes do it, but let’s go slow and make sure we understand it. When asked why, these security folks have a built in scape goat. They blame compliance. Yes, f**king compliance. Yeah, that’s the ticket. We have to go slow because our compliance obligations require it. I laughed out loud when I saw this raggedy old flag being hoisted in George’s article.

Compliance is always the fallback of the security guys who say no. I have seen many examples of this over the years. Two of my favorites are:

1. The security guy who asks the IT manager or C-level guy “how do you look in stripes”, because if you do this without going slow and making sure we are compliant, we could all wind up in the slammer.

Hell we didn’t send anyone to jail for bringing this country to the brink of economic ruin. Do you really think we are going to send someone to jail because by adopting DevOps we might blow a PCI requirement? Can we really do any worse than Target?

2. Walking around a US Army base talking to the security officer I asked him how he was dealing with the compliance issues around securing wireless networks on base. He replied that was easy, he forbid wireless networks on base, so they were in full compliance. Of course as he said this I was watching people unplug their WAPs and throw them under their desks as we walked by.

Yes, security folk can fall back on compliance as the reason for our no, but it is tired and old. The same for the security person who are always promising the next magic bullet. If only they had the latest shiny, new budget busting trinket they could finally sleep well at night. Even if they get their new toy, it only turns up that the bad guys have found a new vector and we need the next silver bullet.

No, I am afraid being the guys who says no or even the people who say OK, but let’s go slow is just not going to cut it. The speed of business today at the enterprise or the startup is anathema to the “go slow” mantra.

On the other hand, DevOps offers a new way. A way to change the way the game is played. For so long I have heard security folk wax poetic about the possibility that security could be built in to the process. Instead of being bolted on after the fact, wouldn’t it be great if it were built in by design? Wow, what a great thing that could be.

DevOps has the potential to deliver on the built in dream. DevOps can make the developer more aware of security and even compliance issues while the code is being written. It can give the security team insight and input into the development planning and process.

DevOps has the potential to break down the silo that separates security from the rest of IT. Just as it brings Dev and Ops together, DevOps can give security a seat at the table. Security can move from being the crazy madman in the back to a respectable member of the team.

This security transformation won’t be easy though. It will demand a change in security’s attitude. First and foremost, we have to change from being the people who say no or go slow, to the people who say “yes we can”. We need to understand that many of these projects and initiatives are going to move forward with or without us. Better to get on the bus than to get run over by the bus.

Next we have to do a better job of communicating our concerns or issues. Falling back on FUD or compliance is not cutting it. We need to communicate the risks involved, how we can mitigate the risk and understand that a business decision will be made as to what level of risk is acceptable and how that risk is going to be managed. At that point we have given enough information for the business people to make a business decision. They make the decision, we get behind it, we don’t bitch bite and we move forward.

Security can’t be the people who say no or go slow. When security starts to say yes we can and this is how we do it, security will be given the seat at the IT table that we deserve.

April 24, 2014
DevOps: Caution Ahead

Despite the continued adoption of enterprise DevOps practices, some organizations, especially those in highly-regulated industries remain cautious about moving forward too quickly.

“There’s no doubt that DevOps brings benefits for some organizations,” says Martin Fisher, director of information security at Atlanta-based WellStar Health System. “However, many pushing for DevOps underestimate the amount of technological and cultural change that is necessary to overcome before moving ahead to boldly, especially among those in security and regulatory compliance roles that are concerned with securing and auditing processes they see as they’re potentially losing control over,” Fisher says. “You can’t audit and secure what you can’t control,” he says.

Eric Cowperthwaite, former chief information security officer at Providence Health and Services agrees. “There are some who want to move too fast. In many organizations the culture just isn’t yet there, especially where they are used to very rigid quality assurance and audit controls,” says Cowperthwaite, currently a VP at Core Security.

It’s certainly not uncommon to find chief security officers in heavily regulated industries who are concerned that the move to DevOps is an excuse to cut corners and move more quickly by cutting out necessary oversight. “You’ll find if you have a developer that’s producing a dozen mistakes a day, if you poke at them about it then you’ll find out that they’re tired of the structure, of the bureaucracy,” he says. “DevOps need to be done in a mindful and thoughtful way,” Fisher says.

Cowperthwaite agrees, and contends that in certain environments DevOps will be – should be – taken slowly. “You have legacy systems and very highly audited and controlled systems where DevOps is just not going to flow for some time,” he says.

It’s not just the cultural shift, but also the shift in necessary for good governance capabilities to keep up, they contend. “Not all systems are geared to handle 200 commits a day, nor do they have the ability for reasonable tracking and accountability for that kind of speed. In fact, there are many legacy systems were it’s not even possible at all,” Fisher says.

However, not that all companies that move toward DevOps are attempting to automate everything – at least not at first. But it is higher levels of automation most certainly seek. A survey recently conducted by automated server management software provider JumpCloud found that many companies are turning to DevOps to automate time consuming activity, such as deployment, patching, user management, log file analysis, and forensics activities.

Some of the trepidation is warranted, to be sure, and some of it may be due to the media and industry focus on the outlier DevOp implementers. “I think, justifiably, a lot of the fear comes from the fact that a lot of the companies that have gotten the most press regarding DevOps are doing things that are radically different than what is going on within the bulk of IT work at traditional enterprises such as banks, healthcare, and organizations like that,” says David Mortman, chief security architect and distinguished engineer at Dell Software Enstratius.

When it comes to enterprises with trepidations when moving toward DevOps, all of the interviewees we spoke with agreed the key is to move slow. But still move and develop the culture and the toolsets slowly. “Pick one system that is separate and use it as a concept,” Fisher advises. “The DevOps ‘true believers’ are given a hall pass to fully run their systems and group in a DevOps model. As they learn and improve you slowly assimilate into the broader environment,” he says.

Mortman sees it in a similar way, but notes that it’s not just about technology and absolute speed, but also collaboration, culture, and relative speed. “How fast is fast for your organization? If you’re only making changes to your big medical records database every 18 months, and you switch to every three months – that’s faster. That’s more continuous deployment than you were doing before. No one has to jump right in,” Mortman says.

“I started working with those sort of DevOps-y concepts back before I was aware of the term “DevOps,” Mortman adds. “I was calling it “Agile InfoSec” and “Agile Ops.” And with that mindset, even the most regulated, paranoid enterprise can embrace DevOps. Start with small, modestly achievable goals and build from there. Collaborate and more tightly integrate the teams.

April 24, 2014