devopsy.kinsta.cloud

Category: DevSecOps

  • From a Commodore 64 to DevSecOps

    From a Commodore 64 to DevSecOps

    We all know the story: a farm, a kid, a Commodore 64, and a modem maxing out at 300bps. A few unexpected phone bills later, and young Ian Allison is figuring out how to game the system so he can keep using his newfound gateway to the world of tech. According to Ian, that is where he began building the foundation of skills for his career in computer security.

    At the recent All Day DevOps conference, Ian (@iallison), now with Intuit, talked about his history of being “that” security guy. You know, the one who thinks developers don’t care about security or deadlines and, really, are just plain “stupid.” But, don’t worry, he is enlightened now and realizes that we all have the same goal: Everyone wants to build a secure system.

    Ian realized that “security doesn’t understand how developers or operations works. Security solves for security, but that leaves everyone else in their own place.” He started his enlightenment when his career path led him to a place called DevSecOps; that is, DevOps where security plays a more integral role.

    Ian pointed out that traditional InfoSec relies on compliance, regulations, appliances and perimeter (CRAP). He then realized the selfishness of his own and his peer’s perspective: Remediation is left up to the developers, the feedback they get are 200-page scanner reports, and it only solves problems for security and compliance. It doesn’t help developers reach their shared goal of a secure system.

    DevOps creates an opportunity for security to get a better view into our infrastructure, operations and development efforts.  DevOps is not only fast, lean and efficient, but when done right, it is also collaborative and empathetic. Couple speed with collaboration and empathy, and DevSecOps can blossom.

    Here is the reality that Ian was facing: Scanners find the absolute bare minimum; bad default configs are a HUGE problem even with SaaS vendors; manual testing can uncover defects that have been hiding for year; and the attackers are more skilled and motivated.

    How do you implement it and make it better?

    • Allow Dev teams to assume the risk of their decisions.
    • No more Security exceptions or signoffs.
    • Security is everyone’s responsibility.
    • Test the crap out of your own stuff like an attacker would.

    At Intuit, Ian wanted to help build stronger bridges between development, operations and security teams.  To do this, he set up a Red Team to:

    • Use same tactics as attackers.
    • Only scope is “Don’t take down production.”
    • Need to adapt and evolve like an attacker.
    • Prove risks actually exist.
    • Should be writing their own exploits.
    • Should have ongoing campaigns that mimic attackers.

    The Red Team started small and lean and focused on the cloud. It worked like an Agile DevOps team, working manually with the use of some tools. In the end, the team found, reported and fixed thousands of vulnerabilities not found by scanners.

    Ian goes into more details and lessons learned in his full All Day DevOps conference session (just 30 minutes). The other 56 presentations from the All Day DevOps Conference are also available online, free-of-charge here.

    This blog series is reviewing sessions from the All Day DevOps conference from November which hosted more than 13,500 registered attendees. Last week I discussed, “System Hardening with Ansible.” Next week, look for my review of an awesome session by Erlend Oftedal:  “There is No Server: Immutable Infrastructure and Serverless Architecture.”

    — Derek E. Weeks

  • Threat Protection Appliances: As Valuable to Security as Your Toaster

    Threat Protection Appliances: As Valuable to Security as Your Toaster

    Nothing in the IT security community is as widely deployed and universally reviled as anti-virus. But, threat detection appliances, including intrusion prevention appliances, application firewalls and advanced threat protection appliances, should be almost as reviled. These appliances are nearly as useless as they are toxic. They do a horrible job finding problems and ultimately create more costs for security organizations than they save by crippling organizations with false alerts.

    Protection is Impractical

    In terms of security, detection appliances have never been effective, since the network makes a lousy location for any kind of security hub from the outset. Few network protection devices can even prevent attacks, much less respond to them, and most appliances are simply used for detection, even those with rudimentary prevention capabilities. In theory, an intrusion prevention device can drop a connection before letting the bad stuff through, but, in practice, that seldom happens.

    In most enterprises, security detection appliances are sitting off to the side, only capable of looking at a copy of network traffic as opposed to the actual network traffic itself. Organizations choose this less-than-ideal setup to avoid the inevitable network latency issues that otherwise occur when a single point of failure such as an appliance gets flooded or has a bug, for example.

    That’s not to say a network appliance can’t respond to an attack in progress in any way. If an appliance determines a particular network connection is malicious, and there is complete confidence in the appliance’s judgment, then an organization can have the device signal something that is in-line, such as a router or firewall. That device can then take action by killing the connection or blacklisting the offending IP address.

    The problem with that scenario is that often there isn’t enough context at the network layer. If a network appliance tries to make fast decisions, it’s going to produce an extraordinarily high number of false positives, making its judgment suspect.  Whenever there’s any doubt about an appliance’s judgment, it shouldn’t be trusted to make decisions whether to take action. Action in such circumstances could affect legitimate connections, which could have catastrophic consequences for the entire business.

    Appliances: What’s Really Happening  

    While old-school devices (e.g., traditional intrusion prevention devices) sacrificed accuracy for speed, today’s more sophisticated appliances do a better job of processing available data, which gives vastly superior results with far fewer false positives.

    The typical “next-gen” security detection appliance (such as an “advanced threat detection” appliance) tries to give as good an answer as possible about whether the connection represents an attack by simulating the end destination of the traffic to predict whether anything malicious will happen. This is done by running a copy of the traffic into a specialized virtual machine, and then monitoring the simulated outcome. If something malicious is found at the destination in the virtual machine, then there’s reasonably high confidence that the connection represents an attack. The goal of this approach is to limit false positives to the point where the organization can respond to every event. Some organization may even be willing to automate that response, by killing the connection if a suspected attack is detected in time, for example.

    While this class of a device tends to reduce noise, it doesn’t eliminate “false negatives,” which allow attacks to sail through without being noticed. In fact, some believe they make the false negatives problem of appliances even worse.

    If an attack works against an actual end user but not in the virtual environment, it could easily be overlooked by an appliance. Imagine a company buys an appliance and configures it to test network traffic by sending it to a virtual machine running Windows 10, but the traffic being tested only runs its proper course if running on Windows 9. In that case, if the network traffic’s destination is a user’s desktop that is running Windows 9, the attack can easily be successful, since the appliance won’t find the problem (because it’s running Windows 10).

    This class of problem was first discovered for network intrusion prevention by Ptacek and Newsham in 1998. While “next-gen” technologies are capable of more accurately modeling the systems they’re trying to protect, this problem will never go away when protecting from the network. Even when companies attempt to force all systems onto a single “gold” image, exceptions tend to quickly come out of the woodwork. For example, users within development and technical organizations often may find themselves in need of software that isn’t on the gold image, or they may need to run older software for testing purposes. Also, the gold image approach tends to only work for desktops.

    Making things worse, the “black hat” community can get their hands on the same appliances and reverse-engineer the virtualization technology and detection techniques it uses. Then, they figure out how to detect whether they’re running on such a device. Once they’ve done that, their real-life attacks will go unnoticed on the appliance, while still reaching the intended target.

    In the “advanced threat detection” appliance space, this problem is prevalent. A dirty secret of the appliance sector is that the majority of detections from these devices are discovered on machines that have already been infected, which are then beaconing out to external command-and-control infrastructure. If instead, these appliances had detected the actual inbound threat, the attack might have been prevented.

    Alert Fatigue: Poor Detection Quality  

    Drowning in alerts is perhaps the single biggest problem IT sSecurity organizations face. Even after all the raw data coming from around the network goes through a best-of-breed correlation and analysis engine, there is still an overflowing sea of information. These besieged engines generate so many alerts that organizations can’t possibly and cost effectively provide proper review for every priority alert they would like to investigate.  As it turns out, the vast majority of alerts are likely to be false positives anyway. Even the best talent can often and easily miss real incidents when pouring through so much superfluous data.

    Imagine the proverbial needle in the haystack, but multiply the number of haystacks so that the entire state of Kansas is covered in hay. Now you can begin imagine the scope of the problem. Sure, you might find an occasional needle or two—especially with some ingenuity. But, there’s virtually no chance that you could tractably look for them all even if you wanted to. That aside, your vigilance is likely to wane for at least a haystack or two, which is all it takes for one devastating needle to puncture your organization.

    Security appliances—particularly more traditional devices—represent the single biggest contributor to alert fatigue by far.  When you’re limiting your investigation to network traffic and trying to get work done as quickly as possible, you don’t have enough context to make good decisions. So, if you want to err on the side of caution, keep in mind that appliances will give you plenty of reason for caution, while uncovering few actual problems.

    If we are serious about eliminating the problem of alert fatigue, we must move beyond appliances and begin managing all of our detection at the machine-level. On a machine we can see what’s happening on the file system, what’s happening in memory, what’s happening in the OS and even what’s happening in the application (for common applications). That’s far more telemetric data to isolate signals and ignore noise (assuming we use all that data wisely).

    Many of the “next-gen” approaches live in a poor middle ground; they try to simulate the machine but generally can’t afford to create an accurate or effective simulation. If an organization is trying to protect a thousand servers, how many appliances would be needed to thoroughly simulate every inch of network and each bit of traffic that crosses? The answer is about a thousand appliances, which is far too cost-prohibitive. Most organizations prefer one appliance to protect hundreds or even thousands of machines, which is clearly insufficient.

    The Future: Will Be Worse

    The standards community is finally embracing the less is more philosophy as it prepares to launch TLS 1.3. This new protocol will ensure there aren’t straightforward back doors to subvert end-to-end encryption. Right now, most security appliances rely on those back doors as a means to provide detection (they need to be able to see the decrypted traffic). The new standard is going to require IT organizations have man-in-the-middle for every hop if they want their security appliances to work. That will add massive latency and expense, so most people won’t do it and appliances will become even less reliable as a result.

    Additionally, the rise of containerization and microservices will exacerbate the situation further, since putting container-to-container traffic on the same machine is well beyond the reach of any appliance.

    Security appliances may seem better than anti-virus, because they occasionally provide some value and don’t add as much attack surface. On the other hand, anti-virus software doesn’t drown organizations in an ocean of spurious reports. Besides, at least anti-virus now has credible alternatives, unlike an appliance. Security appliances are not worth the silicon they’re printed on and should be cursed in the same breath as anti-virus.

    About the Author / John Viega

    John Viega is CEO of Capsule8. Previously, he served as EVP of cloud security provider Silversky, where he led the technological advancement of the company’s cloud-based security suite. Following the successful acquisition of SilverSky by BAE Systems, John went on to serve as EVP of Product. Prior to SilverSky, John was SaaS CTO at McAfee. John is an award-winning author with a half dozen books to his name, including “Building Secure Software” and “Network Security with OpenSSL.”  He also co-designed the GCM encryption algorithm, which is nearly universally supported in the HTTPS and IPSec standards. Connect with him on LinkedIn and Twitter.

     

  • All Day DevOps: System Hardening with Ansible

    All Day DevOps: System Hardening with Ansible

    The DevOps pipeline is constantly changing.  Therefore, relevant security controls must be applied contextually.

    We want to be secure, but I think all of us would rather spend our time developing and deploying software. Keeping up with server updates and all of the other security tasks is like cleaning your home: you know it has to be done, but you really just want to enjoy your clean home. The good news is you can hire a “service” to keep your application security up to date, giving you more time to develop.

    At the recent All Day DevOps conference, Akash Mahajan (@makash), a founder/director at Appsecco, discussed hardening your system’s security with Ansible. In addition to his role at Appsecco, Akash is involved as a local leader with the Open Web Application Security Project (OWASP).

    Misconfiguration

    During his presentation, Akash mentioned the OWASP Top 10 Security Vulnerabilities list, zeroing in on No. 5: Security Misconfiguration. To determine if you comply with the guidelines, No. 5 on the list asks:

    • Is any of your software out of date?
    • Are there any unnecessary features enabled/installed including ports, services, accounts, pages or privileges?
    • Are default accounts and their passwords enabled/unchanged?
    • Are security settings and libraries not set to secure values?

    I am sure no one reading this article still uses the default administrator password, but can we say the same of your peers? Have you gotten around to installing the latest software patches on your server?

     

    Automation

    If a task can be automated, developers automate it. So we should automate our security tasks too, where we can. OWASP provides guidance here, suggesting you should:

    • Have a repeatable security hardening process.
    • Ensure your development, QA and production servers are configured identically but with different passwords.
    • Automate the process to minimize the effort required to setup a new secure environment.
    • Implement a process for deploying all new software updates and patches in a timely manner to each deployed environment.
    • Run scans and audits periodically to help detect future misconfigurations or missing patches.

    This is all part of security hardening, which is, “the process where we identify default configuration present on a system and apply changes that will change the configuration to secure values.” This can be applied to your network, transport, application, and kernel networking parameters.

    Ansible Playbooks

    Ansible is one of the solutions Akash likes to work with, but there are others solutions on the market that provide similar value. Without trying to endorse or evaluate one solution over another, let me share perspectives from Akash’s experience with his toolset.

    Why does he like it? It boils down to playbooks. An Ansible playbook is a codified security document, allowing you to describe the desired state of a system, rather than the specific steps of how to get to that state. As Akash points out, things change—it is better to have the end state described rather than have to change commands when the system changes.

    Other advantages of playbooks include:

    • Playbooks are written in YAML providing us with structure that we can learn and train on
    • Playbooks are text files, so we can use Git for version control
    • Managing playbooks is just like managing any software project
    • Playbooks are infrastructure as code but for security
    • Playbooks consist of roles, a key aspect of security
    • Numerous playbooks are available as open source

    The bottom line is you can—and you should—automate your security hardening process. Your users and other stakeholders will thank you and, most of all, you will thank yourself because you can spend more time on the things you love to do.

    Ansible is just one example of a solution that can be used to automate your security tasks. If you want to know more, Akash goes into further detail on getting started with Ansible in his full All Day DevOps conference session (just 30 minutes). The other 56 presentations from the All Day DevOps Conference are also available online, free of charge here.

    This blog series is reviewing sessions from the All Day DevOps conference from November which hosted over 13,500 registered attendees. Last week I discussed, “DevOps at Massive Scale.” Next week, look for “Operationalizing a Red Team for Fun and Profit,” delivered by Intuit’s own Ian Allison.

    — Derek E. Weeks

  • Ransomware will Double in 2017. Are you Prepared?

    Ransomware will Double in 2017. Are you Prepared?

    A recent report found that ransomware attacks quadrupled in 2016, and are likely to double in 2017. Not surprisingly, a research survey showed that 48 percent of organizations had been hit by ransomware in the past year. And as we highlighted in a recent blog post, next generation cloud applications and databases are quickly becoming an attractive target for criminals.

    Attackers are targeting critical data or operations now, not just files. A few examples from the past two weeks highlight this trend:

    Attackers are getting much more clever and picking valuable targets within a business. It’s only a matter of time before ransomware hits cloud-based applications that run on these platforms. What can you do? Be prepared.

    A layered “onion strategy” means using a set of security tools to spot and isolate malware and ransomware, but that alone isn’t enough. The Department of Homeland Security issued an alert last year highlighting the key role backup plays in defending against ransomware:

    Employ a data backup and recovery plan for all critical information. Perform and test regular backups to limit the impact of data or system loss and to expedite the recovery process. Note that network-connected backups can also be affected by ransomware; critical backups should be isolated from the network for optimum protection.

    As the Texas police department example shows, recovering from a backup won’t work if the backup itself gets infected. A versioned backup of business data and files is essential to recovery. That’s where most backup solutions fall short, especially for next-generation applications and databases that cannot be quiesced to make a consistent backup copy across the database cluster due to their distributed nature. More importantly, given the large amount of data that is processed by these applications, it is critical that there is no bottleneck to data movement in and out of the cluster for an application consistent backup.

    — Jeannie Liou

  • Security @ the Speed of DevOps Survey: Efforts Still Lag

    Security @ the Speed of DevOps Survey: Efforts Still Lag

    Enterprises today are in a relentless race to digitize or die. If they don’t digitize business processes, services or internal workflows and their competitors do, they will be at a steep disadvantage—and perhaps soon an irrecoverable one.

    In more and more companies these efforts are being driven by DevOps. According to IDC, the vast majority of businesses today think—and rightly so—that it is essential for them to put into place the right resources and IT processes to support a digital business model. Unfortunately, about 75 percent of organizations don’t think that their IT organization is actually scalable enough to deliver and support a comprehensive digital business model.

    There are many reasons why so many organizations have a challenging time with their digital transformation efforts. Many of these are cultural: IT commands tightfisted control over all information technologies used throughout the organization, yet being unable to deliver the infrastructure and digital services enterprises need to succeed, for example. Or, others within the organization still are unable to see how rapidly the world is being changed by digital technologies and so aren’t taking the necessary steps.

    Another reason is security. That’s because this rapid digitization of business processes can have a profound impact on how enterprises secure their data, applications and software development and delivery. In devopsy.kinsta.cloud’s inaugural annual “Security @ the Speed of DevOps” survey, we queried 255 security IT decision makers within organizations currently practicing DevOps or that are on a path to DevOps adoption. The survey found that enterprises of all sizes are embracing DevOps, cloud, continuous integration and deployment. But in securing these processes, however, we found vast differences in maturity.

    When it comes to DevOps practices, larger companies are outperforming. For organizations with 5,000 to 9,999 employees, a full 90 percent have either adopted or started DevOps practices. However, only 38 percent of organizations with 500 or fewer employees have started DevOps or widely adopted such practices.

    As I wrote in our report on the survey, smaller organizations also have a way to go with automated security testing. Only 6.5 percent of those with fewer than 100 employees have automated large parts of their application security testing, while 29 percent have automated some of their application security testing. For firms with 5,000 to 10,000 employees, 38 percent have automated large parts of their testing, and nearly 50 percent some of their testing. For those with more than 10,000 employees, these figures are 20 percent and 48 percent, respectively.

    The challenge here, for many organizations, is the same with enterprise security in general: the view that security slows things down. For the business, it’s about getting more digital capabilities out as quickly as is possible. It’s the same for developers. Security too often is seen as standing in the way of this.

    When this conflict is experienced at DevOps speed, there is the risk not only of development slowing down but also security processes breaking. To succeed, enterprises can’t take the same approaches to security as they did in the past. They must find tools and processes that are designed for the highly automated and continuous world of DevOps.

    Have a look at our survey and report to see how enterprises are tackling this challenge today.

    — George V. Hulme

  • A Modern Database Backup and Recovery Checklist

    A Modern Database Backup and Recovery Checklist

    Do you have single points of failure? How do you handle deduplication? These questions and more need consideration when you’re dealing with database backup and recovery.

    Enterprises are quickly onboarding next-generation, mission-critical applications such as eCommerce, artificial intelligence, machine learning and Internet of Things (IoT) on non-relational databases and distributed local storage. However, as organizations move these applications to scalable databases because they are easier to deploy, always available and cost-effective, they also need a robust and scalable backup and recovery solution to protect these modern databases.

    The challenge is that although enterprise adoption of next-generation applications has been rising, the risk of data loss is prevalent. That’s because non-relational databases lack any viable data protection and recovery solution.

    Here is what I mean:

    • Currently, there is a lack of a solution that allows for corrupted data to be removed, replayed and propagated with minimal downtime to customer-facing applications.
    • Reputable studies have concluded that as much as 75 percent of data disasters ultimately leading to downtime are the result of some sort of human error. Because human errors have a big impact on business operations and occur randomly, they cannot be solved by solutions that are offered today, such as native database replication or node-level snapshots.
    • Database replication does not provide point-in-time backup and any-point-in-time recovery, so enterprises cannot go back and fix these errors. In fact, if errors are introduced, the databases’ redundant-node replication can lead to almost immediate corruption across all nodes of the database.

    So, as you begin to explore backup and recovery solution for your next-generation applications, below is a list of items to keep in mind.

    Storage Is No Longer Unified or Shared Storage

    A key architectural transition is occurring where monolithic pools of shared storage are no longer used for next-gen applications and databases. Next-generation databases are scale-out in nature and use commodity storage at a node level, creating a highly distributed storage pool. This distributed storage pool provides both low cost and high performance, but is not conducive to using traditional database backup and recovery tools, which rely on database durability logs (from a shared storage LUN) for a point-in-time backup.

    Eventual Consistency Has a Big Impact on Backup and Recovery

    Next-generation databases are not built around the traditional ACID (Atomicity, Consistency, Isolation, Durability) transaction model. Transactions are a mechanism for guaranteeing these properties; they are a way of grouping related actions together such that as a whole, a group of operations can be atomic, produce consistent results, be isolated from other operations and be durably recorded. Eventual consistency, however, brings forth new challenges to generate a consistent point-in-time view of the database for operational recovery or test/dev needs. Therefore, novel and scale-out data protection technology to address eventual consistency is needed.

    Deduplication Matters at Scale

    With traditional databases, unified or shared database storage, block-level snapshots made image-based recovery and rollback relatively efficient. Distributed storage combined with a scale-out database that commonly involves replication of data rather than joins across tables led to a rapid proliferation of redundant data and increasing storage requirements (three to four copies of every write operation). Effective data reduction at scale is critical to success and economics for effective data protection in this environment.

    Single Points of Failure Do Fail

    Legacy data protection approaches have commonly involved a media server, which presents a choke point for secondary data path and single point of failure. Given the very high ingest rates coupled with the real-time nature of applications deployed on modern scale-out databases, next-generation data protection infrastructure has to be distributed to handle performance (RPO and RTO windows) and resiliency for always-on environments.

    Checklist for Modern Database Data Protection

    • How do you protect your next-generation scale-out databases today? What scenarios of data loss are you worried about?
    • What downtime can your applications or business handle in the event of data corruption?
    • Have you validated backup and recovery for your next-generation databases?
    • Can you prove compliance with regulatory or line-of-business requirements for backup of your critical next-generation infrastructure?
    • Can you recover to a desired or any point in time quickly? What manual effort is required to do this?
    • How much storage is required to store your desired pool of recovery versions?
    • Does your data protection approach scale out with your application and data growth?
    • Can you efficiently refresh your test/dev environment to a smaller or larger cluster size as needed?
    • What happens to your backup solution when a database node fails? Can you still perform backups? How about intermittent network failures?
    • Native formats free you from vendor lock-in and allow for data management services, such as search and analytics on secondary data.
    • Can you recover your data in native format?

    — Jeannie Liou

  • Elasticsearch Ransomware Attacks Highlight Need for Better Security

    Elasticsearch Ransomware Attacks Highlight Need for Better Security

    Recently, reports surfaced that a large number of Elasticsearch servers fell victim to potential ransomware attacks. Ransomware is the type of malware a company doesn’t want on its systems or network. It takes systems hostage, most commonly by encrypting or stealing data, and exposes the owners to blackmail attempts. According to a report by the Herjavec Group, the cost of damages from ransomware was projected to reach $1 billion by the end of 2016.

    A new wave of ransom attacks observed over the last several weeks targets unsecured MongoDB databases. Security researchers Victor Gevers and Niall Merrigan call these attacks a “ransack,” and Merrigan estimates that more than 40,000 databases were impacted in the first two weeks alone.

    Now research shows that Elasticsearch servers, which are configured to be insecure so they can be accessed over the public internet, are being subjected to similar ransom attacks. Victor Gevers tweeted that within the first three days, 2,515 Elasticsearch servers were eradicated and ransomed and 34,298 vulnerable Elasticsearch instances are still open. In the following days, the number of affected servers has risen to more than 5,000. John Matherly, founder of Shodan, tweeted that the vast majority of vulnerable Elasticsearch servers are open on Amazon Web Services (AWS).

    If an Elasticsearch server is hacked, users will find data indices gone and a message that reads: “SEND 0.2 BTC TO THIS WALLET: 1DAsGY4Kt1a4LCTPMH5vm5PqX32eZmot4r IF YOU WANT RECOVER YOUR DATABASE! SEND TO THIS EMAIL YOUR SERVER IP AFTER SENDING THE BITCOINS…”

    The FBI stresses that victims should refuse to pay Bitcoin ransoms, so users might or might not get their data back depending on the security processes they had in place in case of an attack. At this point, it is unclear who is behind the attacks.

    Ironically, what makes these attacks possible is not that Elasticsearch in itself is insecure, because it isn’t. Ransom attacks are possible because these instances have been configured in a way that makes them vulnerable. It’s like leaving the front door open.

    Technology journalist Steven Vaughan-Nichols of ZDNet gave an excellent summary, explaining that, when used by amateurs without any security skills, Elasticsearch is simple to crack. The people deploying instances on AWS clouds are under the impression that AWS is protecting them, but that’s not the case. While AWS tells users how to protect their AWS Elasticsearch instances, users still need to do the work themselves.

    He notes: “The worst thing about this? Just like the MongoDB attacks, none of this would have happened if its programmers had protected its instances with basic, well-known security measures.”

    Elasticsearch is often used in log management, typically as part of the Elastic Stack or ELK, which stands for its main open-source ingredients of Elasticsearch, Lucene and Kibana. Since it’s free, open-source software, ELK is an easy first choice for many. It’s a great, powerful piece of software. These open-source projects are highly active, with thousands of code contributions every month and a growing combined code base of about 2.5 million lines of code.

    Users need expertise to deploy and run it efficiently and safely, though, and that means needing people in an organization with the skills and time to maintain ELK clusters. If these people leave, companies need to have a backup. If they aren’t willing or able to invest in these resources, they are likely to get into trouble, like this latest ransom attack situation illustrates.

    ELK is free software, but keep in mind what Richard Stallman, founder of the Free Software Foundation and the GNU Project, has to say about free: “’Free software’ is a matter of liberty, not price. To understand the concept, you should think of ‘free’ as in ‘free speech,’ not as in ‘free beer.’”

    Whether a company pays for a log management service or runs ELK, one approach isn’t necessarily better than the other. For some companies, it makes a lot of sense to run an in-house-built log management solution based on ELK, or even one built from scratch. For others, a delivered solution may be best.

    Regardless, companies and teams need to carefully evaluate if open source makes business sense and if they are realistically able to properly support the deployment without imposing risks. If these factors aren’t weighed or completed correctly, the number of Elasticsearch ransomware attacks will continue to grow and be a profitable endeavor for hackers.

    About the Author / Sven Dummer

    Sven Dummer is the senior director of product marketing at Loggly. Previously, Sven worked with Yahoo, Wind River (acquired by Intel), SUSE, and Microsoft in product development and management. At Intel, Sven also helped launch (and named) the collaborative Yocto Project, an open-source initiative that enables users to create custom Linux-based systems for embedded products regardless of the hardware architecture. Connect with him on LinkedIn and Twitter.

  • 2017 Security Blogger Awards – Open for Voting!

    2017 Security Blogger Awards – Open for Voting!

    It’s another year, and another awards ceremony to come at the Security Blogger Meet-Up at RSA Conference in San Francisco. It’s time to vote for your favorite blogs, which were selected solely by our esteemed influencers: Ericka Chickowski, George Hulme, Kelly Jackson-Higgins (Dark Reading) and Don MacVittie. We are very grateful for their taking the time to support our bloggers and the event. 

    Before you vote (https://www.surveymonkey.com/r/sbn2017), remember a few things from last year:

    • Anyone can vote in the awards. However, a valid email is required and it will be verified. Only one vote per IP
    • Any nominee caught trying to “game the system” will be disqualified. Please try to play by the rules and may the best blog win
    • If you are going to RSA be sure to register for the Rugged DevOps event on Monday at the Moscone Center. Free to anyone with any type of RSA ticket (even just expo pass holders). You can register for the event on the RSA Conference website. Here is a code for a free expo pass, 1U7DEVOPSXP. If you are already registered for RSA, you can reserve a seat here.
    • You can also participate in our “Security @ the Speed of DevOps” survey here.

    Here are our nominees for the 2017 Security Blogger Awards

    Most Entertaining Security Blog

    Most Educational Security Blog

    Best New Security Blog or Podcast

    Best Security Podcast

    Best Blog Post of the Year

    Best Corporate Security Blog

    Please vote today!

    — Alan Shimel

  • The Secure Developer Podcast: Continuous Security at Chef

    The Secure Developer Podcast: Continuous Security at Chef

    Welcome to The Secure Developer, a podcast about security for developers, covering security tools and practices you can and should adopt into your development workflow. The Secure Developer is hosted by Guy Podjarny, CEO and co-founder of Snyk.

    In this episode of The Secure Developer, Guy talks with Chef CTO Adam Jacob about the role security can play in DevOps and continuous integration/deployment. They cover the differences between baked-in and bolted-on security and how automation with Habitat can change the way developers approach secure coding. Adam Jacob is the co-founder and CTO of Chef, formerly known as Opscode, where he wrote the open-source, automated applications project Habitat.

    This podcast is brought to you by Heavybit, a program that helps developer product companies take their product to market.

    [soundcloud url=”https://api.soundcloud.com/tracks/295763125?secret_token=s-AfcYk” params=”auto_play=false&hide_related=false&show_comments=true&show_user=true&show_reposts=false&visual=false” width=”100%” height=”166″ iframe=”true” /]

    Guy Podjarny: Hello, everybody, and welcome back to The Secure Developer. Today we have an awesome guest on the show, Adam Jacob from Chef. Adam, thanks for joining us.

    Adam Jacob: Hi.

    Guy: We’ll talk about various cool topics we teed up, that I think are interesting with the world of security and the world of DevOps and CI/CD, and some very interesting new package management, build system capabilities coming out of Chef that I think are very relevant to the security play.

    Before we dig in, for the few of you that might not know Adam Jacob or Chef, Adam, do you want to give a quick intro of your background?

    Adam: Sure, I think there’s probably more than a few who have no idea who I am. I’m Adam. I wrote Chef, originally. I’m the CTO at Chef, and I wrote a thing called Habitat not that long ago that does application automation. And that’s sort of new stuff.

    Then, mostly what all of that really boils down to is, I spent the last 10 years going around talking to big web companies like Facebook and Google and Yahoo, and I’ve also spent a bunch of time with startups. And then I’ve also gotten to go see really large enterprises ranging from giant banks to insurance companies, retail companies like Nordstrom, Wal-Mart.

    So I get to just travel around and see what everybody is doing and see what they’re worried about and try to help them get better in terms of the time it takes or the speed to deliver, or their organization or their culture. So a bunch of it is software, but a lot of it is just helping people sort of understand how better to build their organization.

    Guy: Yeah, I guess there’s always this conversation about whether DevOps and, well, continuous deployment maybe is a little bit more specific, but DevOps is really more about the tools and more about the people and it tends to be a consensus that it’s more people.

    Adam: It’s both, right? Like, if you have a great culture, it’s really easy to say that you want to have a good culture. It’s really easy to be like, “Oh, I want to empower people,” or, “We want to streamline a process,” or whatever. It takes nothing. It’s just words.

    Usually, it’s the technology that reinforces those cultural behaviors that hold you back.

    So, a good example is, “Oh, I want to do continuous delivery, but we use a terrible source control system that makes it almost impossible to do effective continuous delivery. But because that’s the source control system we use, we’ll never change.”

    Therefore, which one’s true? Do you want to do continuous delivery, or do you love your bad source control system? So there’s this reinforcing circle that hides inside there, and I think you see that all the time in security, too, that same circle and that same behavior.

    Everybody says, “Oh, we want to be secure.” I was working with a bank who I won’t name, and I was on an engagement for a couple weeks.One of my first questions that’s going to help them do continuous delivery, and they wanted to, their first target was, to harden the operating system.

    I asked them, “Okay, do you have the hardening spec? Do you know what you want to harden?” And they were like, “Oh yeah, yeah, of course we do. We’re a global bank.” And I’m like, “That’s amazing, great! I would love to see that.” And they’re like, “Oh, we’ll have it for you on the day you arrive.”

    I left three weeks later, and they had never found it. And what they had realized was it didn’t exist. Everybody thought someone else had built that, and that it was someone else’s job, and no one actually did. It was just this loose conglomeration of stuff that theoretically they were supposed to do, but no one actually could track or knew. And that’s a global bank. It probably should matter.

    Guy: I think sometimes it’s about the mess or the fact that tools can help surface information and hold information in a way that is accessible. And sometimes it’s just the sheer obstacle for many of these more complex topics when you talk about deep ops topics around how containers operate or how some machines are orchestrated.

    Or in the world of security, when you talk about deeper security, understanding about what’s an attack, what’s not, and that sort of a constantly moving landscape. Just fundamentally, you have to have the tools. You cannot overcome those problems by sheer education. And at the same time,

    if you have tools and people don’t know how to use them, or they have no understanding of what they’re for, then you would eventually not achieve what you’re aiming for.

    Adam: Context is everything, right? Tooling is great, but it’s not enough. One of the things that we do at Chef and in the security world, is we have a thing called InSpec, which is a language for letting you describe security posture and code.

    So you can say, “This particular machine should have this particular policy. That policy means that port 25 should be open, you should be able to auth this way, you shouldn’t be able to auth this way.”You should be able to talk about packages being installed, you should be able to talk about all of those things and then relate them back up to the actual security line items and talk about their severity.

    We care about this thing because of this piece of HIPAA, or we care about it because of this piece of CIS, or whatever. And those tools are great because they sort of combine the documentation of, this is what the standard is, with the check that actually is executable that says, “And are we meeting the standard, yes or no?”

    I think whether it’s InSpec or tools like InSpec, when you think about that operational part of security and how it applies to those large enterprises, especially large enterprises, but the big web, too, more and more it’s becoming that policy is executable.

    So that conversation between security and developers and operators becomes a conversation around code as opposed to a conversation around documents, and controls, which I think is really the conversation most people have now.

    Some security guy wrote a control and then here’s the list of people that say that they can validate the control. But is the control any good?

    Guy: Is it actually happening?

    Adam: No. I mean, it’s not, right?

    Guy: Yeah.

    Adam: And that’s true everywhere. We wrote a control. Here’s the list of 10 people that you could go interview that know how to do that control. Okay, what about the other 10 people who could also do that, that aren’t on the interview list that wasn’t updated on the last year? Do they always do the procedure the right way?

    Do they always make that process right? And of course, the answer is no. But we just sort of let it ride because the auditor passed.

    Guy: Yeah, it’s the notion of protecting yourself from audits, not from attacks.

    Adam: Exactly!

    Guy: It becomes sort of increasingly there. I love that notion. So with InSpec or, in general, with security, one of the key challenges is that it’s invisible. So if you’re not doing it, you really have no immediate indicator to the fact that it’s not happening.

    The user experience, if you will, of not monitoring, not watching for a certain vulnerability, and the user experience of watching for it and having nothing happen, is the same thing, which is, nothing happens, right?

    Adam: Yeah, which is nothing.

    Guy: Which is good news. So I think anything that helps define and articulate the controls that are supposed to be happening, giving you some mechanisms to understand that the action has been taken so you know that InSpec, the locking or whatever, say, if we use a simple example, the limiting of ports being open, that has been explicitly articulated and explicitly tracked by the tools here, and you can get informed that it has happened.

    We see this in Snyk a lot. We watch projects for vulnerabilities, for vulnerable dependencies, and then you have really, again, it comes back to the fact that you have the same user experience. If you’re not watching a project, or if you’re watching a project for vulnerabilities and none happened there, and we have a lot of these conversations with users today about, “How do you want to hear about it? How do you want to know that you’re watching it?”

    And today, we remain fairly simple. We just remain in this ongoing report that just sort of shows you, “Hey, you’re monitoring five projects” or something, and they have those things.

    Adam: Right, it shows that it reacts often, or whatever, and we can measure change.

    Guy: But it’s great to sort of increasingly have those and have some requirement, whatever, in your Chef and InSpec, and that defines it as well, project X is being monitored for X, Y, Z, so you know it’s being enforced.

    Adam: Right, and then you stick that in the pipelines.

    When you think about continuous delivery and security, security and the validation of that compliance is part of the process by which software goes into production.

    And it’s part of the way that software gets maintained once it’s there, and it’s part of that build process. You bake it in sort of throughout the SDLC, instead of it being a thing that happens at the end.

    And that’s one of those things that’s obviously a good idea. So, as soon as you hear it, you’re like, “Oh, of course, security should be baked into the process, all through the whole thing.”

    Guy: The whole built-in versus bolt-on.

    Adam: Yeah, right? Sort of, duh. But actually, doing it is actually completely–

    Guy: A whole different ball game.

    Adam: A whole other ball game. And the thing that we really came to realize, especially with InSpec, but just sort of in general, is that if you can’t figure out how to manage that security posture the same way you manage the rest of what you do, it’s really difficult to then tell a software developer that it’s their responsibility to ensure that that posture is good or bad. Because, sure, they can make that they write “good code.” You can’t hear the air quotes, but m I was making little air quotes in my brain, probably.

    Guy: I can attest to that.

    Adam: It’s good, but you can’t really ask them to understand the posture of what it’s going to be like when it’s deployed.

    Because the distance from a software developer making a decision to a software developer talking about how that software should be in production, and what it’s posture ought to be, is so vast. And their ability to influence it is so low that it’s really difficult to come back to them and be like, “Oh, this was your responsibility.” Like, clearly it was on you.

    And when those tools give you the ability to talk about it as code. They allow people to participate, you can code-review them. You can have security people audit the code as opposed to audit your documentation, and then have those things as a living piece of that deployment model, I think everything gets better and it gets a lot more secure.

    One hard part of doing things that way is that it’s very not the way that most security regimes are set up right now. If you go talk to a security officer and you ask them, “Hey, could we remain compliant to whatever the standard is, HIPAA?” If everything we did was continuously delivered, nine times out of 10, the answer is just a flat no.

    And what’s interesting is that there is, one out of 10, where that security officer is like, “I don’t know, maybe. Tell me more about how that would work.” And I think if I roll my clock back three years, it was 10 out of 10 that we’re telling me no and that was crazy. And now it’s like nine of 10. And I predict in six months, it’ll be six of 10. And then in two years, it’ll be like talking about, “Should you be doing continuous delivery or should you be doing Agile?”

    Guy: It comes back actually to the commentations sometimes or even that document or something that big bank did not have. Like, these things eventually come back to the documents and to the guidelines. So at the end of the day, you go down the compliance route and you actually have that in the compliance document that it says, well, if you are using continuous delivery, then to be HIPAA-compliant, you need to ensure that you’re doing these things as opposed to just sort of having the goal in it.

    Many of these regulations have the same flaw, the same notion that says, “Well, you have to do these actions,” as opposed to, “You have to achieve these goals.” So they give the vague goals, but then everybody prescribes to whatever recommended actions that have been prescribed, because that makes passing the audit the easiest.

    Adam: Yeah, and I mean,

    the relationship between the auditor and your security posture is pretty tight.

    Like, the real test of your security posture, in most cases, is the auditor. Not like a pen test, or people actually trying to break you. I mean, they are, but you’re not doing it proactively.

    I think when you think about those CD pipelines, the idea that they’re applied continuously, that as applications change, that you’re reapplying the security posture to see if that application has done something that violates that posture, and that when you change the security posture, you’re re-validating the applications and you’re doing it sort of throughout the whole cycle, it’s super powerful. It’s what people are starting to be able to do. I think you don’t see it a lot yet, but it’s sort of the future.

    Guy: I think that’s interesting to understand, the potential versus what’s happening, right? High-level continuous deployment and your whole infrastructure’s code, the fact that you’ve prescribed it, built it in, allows you to get predictability. You know what’s where, and you know that a certain test or a certain enforcement has been done to the extent of bugs. But at least, not as human error.

    On the flip side, it requires security auditors to change how they behave. It’s not just about that compliance, it’s also the fact that today, many of these security audits, you gave the example of a security auditor reviewing your code, are done as gates.

    They’re done as a way that says ‘stop here,’ which is the antithesis to continuous.

    The whole notion of continuous is just, you roll out. It’s okay to pause for a moment, but you can’t stop. You can’t accumulate a backlog, because otherwise it’ll deteriorate, unless you automate that.

    Adam: Yes and no, right? There’s continuous deployment, and then there’s continuous delivery,and they’re not quite the same. So yes, in continuous deployment, there’s nowhere to pause. I think in continuous delivery, there is.

    In continuous delivery, the idea is you’re shipping when you should be able to ship, any time that the business required shipping or it made sense to ship. And that’s different than saying, “We just ship every time you commit,” right? And so in continuous delivery, I think there’s plenty of space to say that, “Hey, this project, in order to ship, requires a security review.” And that process can still be continuous.

    The question is, that should be the only gate between you and getting to production. So if they say yes, could you ship? And then the question of course is, is that true for every commit? Like, could you ship today, if today was the time? The answer to that is very rarely yes, and so that’s where the difficulty comes in.

    Guy: Yeah, and I think it’s an interesting comment. I accept the delta between continuous delivery and continuous deployment, but it seems to me that from security route, well, first of all, even from a quality perspective, one of the values of continuous, both delivery and deployment, is the fact that you spot errors when they’re small, at any given time.

    So definitely, for continuous deployment, when you ship every small code change, whatever, at some low resolution, when a problem occurs, it’s much easier to pinpoint to what was the issue, the source of the problem.

    That value proposition is pretty compelling in security as well, right? If you are looking for a security flaw, being able to, only looking at it within the range of whatever the 100 lines of code to change versus the 100,000 lines of code or 100 million that existed, is very valuable. If you accumulate those changes and you need to now do the security audit, I would argue you’re not actually ready to deliver in that spot.

    This podcast is brought to you by Heavybit, a program that helps developer-focused companies take their product to market. Throughout the program, founders work on developer traction, product market fit and customer development. If you like what you hear, be sure to subscribe to Heavybit’s Podcast Network on iTunes.

     

    — Malia Powers

  • What the Current DDoS Landscape Means for DevOps

    What the Current DDoS Landscape Means for DevOps

    Businesses, governments, hospitals, schools, charities and even individuals—they’re all the same to a DDoS perpetrator. If you have a website, it’s likely to be targeted by a distributed denial of service (DDoS) attack at some point.

    DDoS has the potential to shut down your site for days and create havoc with an organization. They can be expensive and—most worrisome—can taint your reputation and discourage users and customers from ever returning if your site is unavailable on any given occasion.

    Perpetrators have been known to exploit network packets and all types of vulnerabilities, including writing custom code dedicated to knock down a specific application or service, to overburden systems or stop them outright.

    Attack Motivation

    There are plenty of threat actors ready to launch a DDoS barrage. Some of them include:

    • Hacktivists using DDoS to express their discontent with businesses, governments and individuals.
    • Cybercriminals relying on pre-made scripts and tools to take sites down. Some instigators are simply looking for a way to vent their anger or frustration. Some may use commonly available DDoS-for-hire websites instead of running the attacks from their own network.
    • Extortionists who blackmail sites, demanding money in exchange for stopping (or not carrying out) a DDoS threat.
    • Business competitors seeking ways to exclude rivals from significant events (e.g., Cyber Monday), or attempting to completely shut them down.
    • State-funded threat actors engaging in cyber warfare to silence critics and opponents. They also can target civic infrastructures to cripple opposing countries.
    • Attackers who attack just because they can, or for no obvious reason, other than test their ability to carry out an attack.

    DDoS Types and Trends

    But DDoS events don’t just disrupt service. According to “Cloud Security Alliance Guide to Cloud Computing,” hackers are also using them to steal information and infect computers for a variety of nefarious purposes.

    In addition, DDoS assaults are growing rapidly in both number and volume. International Business Times states they’ve become more commonplace because readily available tools and cheap online services let anyone aim an attack against a company or individual server.

    DDoS attacks can be divided into three types, with numerous (and unique) variations within each.

    Volume-based attacks saturate the bandwidth of a site. Imperva Incapsula, a cloud-based security and acceleration provider, faced the largest such attack on its record in 2Q 2016, peaking at 470 Gbps. Like many other complex, high-rate assaults, attackers used small payloads to achieve a high packet forwarding rate—a dangerous new tactic that has become common.

    The main purpose of such attacks is to take down mitigation services by sending out a rapid burst of packets at a rate many anti-DDoS appliances can’t handle.

    Protocol attacks (aimed at the OSI layer 4) consume server resources such as firewalls and load balancers. Network layer attacks have grown in size, number and sophistication. Those using multiple vectors have climbed to a record-high 36.1 percent, reports Incapsula. On average, it mitigated a 50+ Mpps attack every three days in that quarter.

    Network layer attack duration increased in the same period, with 13 percent lasting for over an hour. The longest persisted for more than 10 days in a row. While most are in the hit and run category—using short bursts launched against the same target—an uptrend points to the prevalence of events lasting more than six hours.

    Application layer assaults (targeting layer 7) are comprised of seemingly legitimate HTTP requests, attributed to bad bots that have also grown in sophistication. An ongoing salvo of requests originating from numerous masked IP addresses can bring your web application down in no time, by creating stress on the web servers, database servers or other elements of the web application.

    The largest such event mitigated by Incapsula in 2Q peaked at 108,288 RPS (requests per second). The longest ran its course over 67 days, while 59 percent lasted less than 30 minutes. The company attributes this to an increased number of “casual” offenders.

    Examining Risk

    The Open Web Application Security Project (OWASP) offers a brief look at risk assessment:

    • “… inadequate resources, requires attention if system architecture was not designed to meet traffic demand overflows … left unchecked, [it can] result in DoS symptoms absent an actual attack.
    • “… perhaps the largest risk factor is not technical … An organization should avoid taking action that can make them a target of a DoS attack unless the benefits of doing so outweigh the potential costs or mitigating controls are in place.
    • “Other risk factors may also exist depending on [your] specific environment.”

    The first item above can apply to any website. While one might think that the second might only be applicable to political entities, what about ecommerce sites based on competitive pricing? Unless strong DDoS defenses are in place, such a site won’t last long in today’s ultra-competitive digital world.

    So what can an organization do about fending off DDoS attacks? With assaults becoming both easy to launch and more sophisticated with each passing day, it’s imperative to keep up to date regarding the evolving threat landscape.

    Securing Your Apps

    Using software and plug-ins for which the latest security patches have been applied is a great start. Penetration testing is a highly recommended critical step before going live. Here, OWASP offers a complete online guide to assist you in your efforts.

    OWASP also offers several examples showing where code vulnerabilities may have been overlooked, ranging from user-specified object allocation to locking customer accounts.

    Once your app has gone live, monitoring site traffic to benchmark volume and visitor types helps ensure its reliability, as unwanted traffic can be detected and quickly addressed. Assessing traffic flow summaries is a start, followed by such tasks as examining IP source geography and unique source IPs hitting your site.

    But data analysis is only a start. SANS Institute offers this guide to help you learn more about successfully mitigating a DDoS attack.

    Most importantly, your operations team can create a response plan to minimize the impact of an assault. An effective plan includes procedures for your customer support and communications teams, as well as keeping CxO executives in the loop.

    Choosing the Right Mitigation Option

    Find a mitigation strategy that works best for your specific business needs. Planning includes prioritizing your concerns and examining the benefits of various mitigation options against your security budget. This is where it’s potentially more cost-effective to engage specialty security services (and their dedicated teams) rather than to try to “roll your own.”

    Ensure the DDoS protection you currently have offers the scalability and security capabilities needed to keep your site and server from crashing in the event of an attack.

    About the Author  / Ben Herzberg

    ben-herzbergBen Herzberg is security research group manager for the Imperva Incapsula product line at Imperva. Ben’s a  developer, hacker and technical manager, deeply interested in different technologies and focused on information security. He enjoys developing something new that just wasn’t there before, or solving a puzzle in a different way. Connect with him on LinkedIn and Twitter.