devopsy.kinsta.cloud

Category: DevSecOps

  • How Secure is Open Source for DevOps? 5 Considerations

    How Secure is Open Source for DevOps? 5 Considerations

    Open source libraries and frameworks have important roles to play in a DevOps culture that emphasizes shorter development life cycles, collaboration and innovation. However, it’s vital not to neglect the security of these open source components.

    This article discusses the use of open source in DevOps and gives you five things to consider in terms of the security of any open source project your teams might want to use in your software builds.

    Open Source and DevOps Overview

    Proprietary applications, many of which are developed within organizations that have adopted a DevOps approach, feature an average of 257 open source components per application. Furthermore, the average percentage of codebases that are open source within proprietary apps stands at 57 percent. It’s clear that open source fits well with DevOps in how it can help deliver quality software that can be rapidly updated and improved.

    In addition to the increasingly popular use of open source libraries and frameworks to deliver ready-made software components, quite a number of tools used in DevOps are actually open source. Examples of open source DevOps tools include Docker and Kubernetes.

    It’s crucial that development teams focus on security when using open source code, components and tools. Several of the most high-profile IT security and software breaches of recent times have resulted from vulnerabilities in open source components.

    To adequately combine security, open source and DevOps, there is a pressing need to shift left and secure the software development life cycle (SDLC), starting from the earliest stages of development.

    DevOps Open Source Security Considerations

    The Need For Security Automation

    An important driving force in achieving the aims of DevOps is to automate as much as possible. DevOps automation strategies include the use of technologies such as virtual machines and containerization to repackage applications into reusable blocks, many of which contain open source code. This high level of automation dramatically shortens the SDLC.

    However, because of this automation and the pace at which code changes and updates occur, security teams are easily left behind, and they might miss out on identifying open source vulnerabilities. InfoSec teams need to figure out ways of automating several of the most important security procedures, such as configuration checks, code analysis and vulnerability scanning. By introducing greater automation into security checks, it’s less likely that DevOps practices will lead to releasing software containing vulnerable open source components. Faster time to market combined with high-quality, secure code is the ultimate goal.

    Securing Open Source Using Open Source Tools

    Given that open source code forms the majority of the footprint of modern proprietary codebases, it makes sense to focus on the security of these libraries and frameworks first. Somewhat ironically, open source tools provide a good way to improve open source security in DevOps.

    For example, teams can use open source software management programs such as JFrog Artifactory to create binary local repositories of open source software libraries and code. These repositories give teams a single place from which they can use the latest versions of open source code that they have deemed to be secure.

    OWASP dependency-check is another useful open source tool for security teams that checks which components an application uses and whether such components contain known vulnerabilities.

    Incorporate Open Source Code-Checking Tools Into Development

    Part of the idea of shifting security to the left involves developers overcoming the inherent tendency to focus solely on application functionality without considering security. While training materials exist that teach the basics of secure apps, developers might find the material somewhat esoteric, particularly if it’s targeted toward InfoSec teams.

    One way to overcome developer resistance and shift security to the left is to integrate some open source code-checking tools into development environments. Tools such as Brakeman or Rubocop are freely available, and they can rapidly check code for common vulnerabilities.

    Hackers Targeting Open Source

    One of the difficulties with the increased use of open source is that hackers and cybercriminals are always aware of components that contain vulnerabilities. The hackers can then use this knowledge to specifically target companies developing software using those components.

    Cybercriminals know that DevOps emphasizes faster time to market, and they are always lurking to find those organizations that have become lax in their IT security checks. The Equifax data breach happened precisely because the company used a vulnerable version of the open source Apache Struts framework. The Equifax case exemplifies the potential danger of open source components in organizations where DevOps and InfoSec are out of sync.

    The Need for Policy and Governance 

    For DevOps and open source to work together to deliver secure software, there is a real need to create dedicated open source policies by DevOps organizations. When developers are given free rein to use open source libraries and frameworks without monitoring or documentation, vulnerabilities creep into applications and future trouble awaits.

    The first step is for organizations to recognize their reliance on open source before forming a governance policy that stipulates how to manage open source inventories to reduce security risks.

    Wrap Up

    Going forward, it’s clear that the open source model offers many advantages to DevOps teams in terms of achieving their goals.  Additionally, several open source tools can even enhance the security of the applications developed in DevOps. However, it’s prudent to understand the security concerns that lie at the heart of open source and shift security left so that any vulnerabilities can be found as early as possible in the SDLC.

    — Limor Wainstein

  • 3 Tips to Build A DevSecOps Organization

    3 Tips to Build A DevSecOps Organization

    For organizations undergoing digital transformation today, modernizing the existing environment can present serious challenges when it comes to security. Whether you’re dealing with a transition from legacy to cloud or hybrid models, or shifting from ITIL and waterfall methodologies to DevOps, the increasing number of technologies and complexity mean you need to rethink your security posture. The traditional approach to security—that is, a static strategy that largely involves patching on piecemeal measures at the 11th hour—is simply not a winning formula anymore, especially when dealing with requirements such as privacy compliance or contractual agreements.

    DevSecOps represents a fundamental shift in which real business needs drive a dynamic, living/breathing approach to security based on continuously changing requirements. To evolve from DevOps to DevSecOps, an organization must focus on integrating security into the very fabric of the software development cycle, and work to increase intelligence, situational awareness, and collaboration.

    Here are three critical ways to consider to ensure your DevSecOps strategy is up to snuff.

    Create a Single Source of Truth

    When you have multiple teams trying to work at breakneck speed, having one absolute source of data is essential. Gone are the days when we could rely on static spreadsheets that lived locally on this or that person’s computer, and even communication mechanisms such as email are too manual and out of sync to be trusted. What’s more, it’s impossible to draw meaningful correlations and map trends if your data is sitting in silos across your organization.

    Creating a single source of truth will ensure the greatest accuracy of information for everyone. You need to pinpoint where your data is coming from, how it should be collected and how it should be shared. You’ll want to integrate your full tool stack and workflow, and harness automation to streamline hand-offs between collaboration tools, system updates, chatbots and more.

    Shift Left: Secure By Design

    Relying on firewalls and antivirus as your primary security measures is a bad, bad habit. The key is instead to shift left of these elements and work to embed privacy from the start. This is the new age of security, using a risk-based approach instead of a reactive one—that is, identifying what needs protection, why it must be protected and how you will do so. It’s also understanding that security should not be just an external threat perspective, but also having visibility into what’s happening internally.

    The key drivers of your secure-by-design approach for DevSecOps should be privacy compliance, contractual obligations and threat models, both known and emerging. The goal here is to be proactive and avoid spending valuable resources to address threats or vulnerabilities that could have been prevented. And in case you didn’t already know, properly written, patched and documented code accounts for about 80 percent of security!

    Understand Your Environment

    If your organization has embraced DevOps, then you’re likely aware of necessities such as process, collaboration and automation. However, these can sometimes come at the expense of other important things, including privacy and security. A lot of this is due to lack of oversight and poor visibility into change management.

    As organizations accelerate their adoption of cloud services, threat vectors are ever-expanding. As such, you need to have complete situational awareness of your organization. You need to know what to monitor for and when, and this cannot be limited to the events directly related to security. Instead, focus on extending your perimeter of knowledge beyond your DevOps pipeline and ensure you’re monitoring everything from operating system logs and directory systems to DNS and servers. Without all of this context, there’s simply no way to correlate security incidents with other data from your IT environment. This is the information you need to document processes, workflows and playbooks, and ensure your teams can communicate and collaborate rapidly to address issues before the business is impacted.

    Remember, when it comes to the ultimate big-picture goal of DevSecOps, it’s always about minimizing the financial impact to your organization. Whether we’re talking about your reputation or lost time and resources, the bottom line is dollars down the drain. And the data is compelling so far—in its “State of DevOps 2017 Report,” Puppet found that high-performing DevOps teams experience a 96X faster MTTR from downtime and a 5X lower change failure rate compared to lower-performing teams.

    This article was co-written by Domnick Eger.

    — Robert Hawk

  • Are Developers Your First Line of Security Risk or Defense?

    Are Developers Your First Line of Security Risk or Defense?

    Every organization on the planet, whether public or commercial, is facing an ongoing challenge: to exist in an increasingly digital space. Past products must transform to meet the digital expectations of their customers, and new products must be built with “digital-first” frameworks in mind. After all, if they fail to meet these expectations, there’s almost certainly a competitor ready to swoop on the opportunity.

    So, what does that mean for all these businesses, old and new? It means they are employing teams of developers to create line after line of the code that forms their digital product or service.

    In this fast-moving, high-pressure environment, CIOs have become the new innovators, and are in positions of immense influence. However, they face a strong demand to meet rapid speed-to-market goals, enhance product quality with each update and increase flexibility. This has led to complex, often fragmented development teams whose primary focus is on rapid development of features and functions, without consideration of the vulnerabilities they could be creating in the code they are so quickly churning out.

    If our data is to remain safe, we must implement a new way of working. Just look at the catastrophic damage Equifax faced following its recent breach: a disaster that occurred as a direct result of insecure coding practices.

    CIOs and CISOs need to consider carefully whether their development teams are in fact the first line of data breach risk in the organization or the security champions, their company’s true “first line of defense” against cyberattacks. If they’re not the latter, they certainly could be.

    Secure Coding Checklist

    Where does your own dev team fit? I have created this Secure Coding Checklist to help CIOs and CISOs identify whether their development teams are truly equipped to be secure coding champions, helping to innovate with faster, better and safer code (or, indeed, whether you’re in need of a more robust security program):

    1. Is there acknowledgment at your executive C-level that traditional network security is no longer enough?

    These days, securing the network layer using traditional security measures is simply not good enough (and, let’s face it, is rarely successful anyway) even against semi-professional hackers. Among many consistent reports, Verizon’s “2017 Data Breach Investigations Report” cites that 35 percent of data breaches today are caused by web application vulnerabilities.

    Web application security is equally as important as network security; ignoring fundamental, protective AppSec measures could leave you well and truly open to a breach.

    2. Are you considering security from the start?

    The current approach to application security utilizes tools that focus on moving from right to left in the software development life cycle (SDLC). This, by design, supports a detection and reaction outcome. This means that security teams search for and detect vulnerabilities code that is already written, then react to fix them. According to the National Institute of Standards and Technology (NIST), it is 30 times more expensive to detect and fix vulnerabilities in committed code than it is to prevent them as they are written in the IDE. This doesn’t even take into consideration the production delays, double-handling and time spent in remedying any issues. Secure code champions start at the extreme left of the SDLC, with a focus on continually training their developers in secure coding, so they can be the first line of defense in their organization, preventing common vulnerabilities from occurring in the first place.

    3. Do you actually make the effort to build security skills, or are you just feeding knowledge?

    The vast majority of security training solutions (online and in-classroom) focus on building knowledge instead of building practical skills. For developers to thrive and engage in writing secure code, they need regular access to hands-on learning that actively encourages them to learn and build their skills in a real environment. They need to learn about recently identified vulnerabilities, in real code, and be able to work in their own languages/frameworks. This learning experience should help them understand how to locate, identify and fix known vulnerabilities in the code they are actively working on in the course of their jobs.

    4. Are you measuring your secure coding skills with real-time metrics?

    A vital step in ensuring a security-first mindset is being built within a development team is to collect and review evidence. Metrics seek to prove to the developer and their organization that their hard work is paying off, and secure coding skills are improving. You can’t improve what you cannot measure. There should be relevant assessments, and these should help identify the progress of your development teams in real-time as well as benchmarking their secure coding strengths and weaknesses for continuous improvement.

    5. How sure are you that any outsourced vendors are actively using secure coding techniques?

    Many organizations decide to contract out development work to other development houses. Whether they exist on- or offshore, their general secure coding abilities and practices are a relative mystery. In the best case, the only form of assurance an organization will receive relating to security is a statement in the contract requiring the deliverable to be “secure.” Few companies actually take the step to verify the skills of these development houses upfront, thus winding up with software that has been built without following sound secure coding practices. Worse yet, if the company is not aware of them, it risks going live with a vulnerable application.

    The most common scenario is that any vulnerabilities get picked up by dedicated security specialists (that come at a high cost) and you are faced with a delay of your go-live date, and possible contractual discussions on who needs to pay for fixing these security weaknesses. However, if you’re smart upfront and assess the application security skills of the developers who are going to build your next application, you save a lot of potential delay, frustration and cash.

    6. Are your developers aware of the most common security weaknesses?

    More than 85 percent of exploited web application vulnerabilities are attributed to just 10 known vulnerabilities—the OWASP Top 10. At a bare minimum, your application security training must cover these, in addition to many more vulnerability types. The training challenges your developers contend with must be actively revised and updated on a regular basis, with new challenges for either new coding frameworks or new vulnerability types.

    7. Do you have in-house security champions?

    Every developer-heavy organization should invest in a security champion—someone who is going to take responsibility for a security standard within the dev team. Their purpose is to be a support point of contact for anyone with questions around security, as well as become the chief advocate for following security best practice.

    8. Have you invested in tools for your developers to make secure coding easier?

    If your organization is one in which agile development is practiced, or indeed you are faced with frequent updates to company-built applications, automating parts of your security is one of the only ways to keep up with the frantic pace and volume of work.

    There are tools available at each stage of the SDLC that will serve as advisers, quality gatekeepers or detection tools. Ideally, you should have IDE plugins that focus on certain types of security bugs and act almost like a “spell-checker” while developers are writing their code. There are others that integrate with the build process, detecting certain types of weaknesses as the code is being submitted to a code repository.

    In addition, some tools run automated tests over the code, simulating attempted hacking once the software is in production. All have their own set of benefits and challenges, and none can provide a catchall, 100 percent guarantee that no security threats exist within the application. The No. 1 preventative measure you can employ is that the earlier you can capture the weakness, the faster and cheaper it is to remediate it with the least impact on your business.

    CISOs and CIOs are essentially forced to quickly and aggressively build their enterprise agile capabilities. As such, secure coding skills will be a weapon—not a hindrance to—innovation, and foregoing them could spell the absolute destruction of company reputation and data. Think twice before you skip this critical capability.

    So, how did your organization fare against this checklist?

    — Pieter Danhieux

  • 10 Years of DevOps—Transition, Growth and the Future of AIOps

    10 Years of DevOps—Transition, Growth and the Future of AIOps

    When software development became a cumbersome task of repetitive coding, testing and glitches, organizations had to come up with a solution.

    The solution? An organized approach that would not only eliminate the endless cycle of repetitive tasks but also would establish proper coordination and execution in the software development life cycle (SDLC).

    That very systematic approach with fewer glitches, more production and exemplary coordination today is called DevOps.

    But where did this wonderful process start? How did it make its pathway into our the cycle of IT industry?

    The Beginning

    It all began one day when Andrew Clay Shafer scheduled a session, “Agile Infrastructure,” at the Agile Conference in Toronto.

    However, the DevOps model back then didn’t seem to make much of a difference to those associated with software development. As a result, Shafer closed the session. But something struck developer Patrick Debois that day, who was presenting a paper on “Agile Infrastructure and Operations” in the same vicinity. Shafer’s theme seemed to draw his attention and he went straight to meet Shafer.

    A long talk in the hallway that day ignited the fire of DevOps, which started with Shafer and Patrick three months after their first meeting.

    Together, they started the Agile Systems Group, with an aim to foster greater collaboration between developers and operators.

    Today, the DevOps process continues to survive on the principles of increased collaboration and automation to build, test and release software faster.

    Mobile Device and DevOps Adoption

    Over the years, DevOps began to grow and pick up fame. And with the emergence of mobile phones, the trend picked up rampant speed.

    According to a market intelligence company App Annie, global economic activity concerning mobile applications will be worth USD $6.3 trillion by 2021, five times 2016’s revenue.

    Creating software for mobile devices requires an elevated software development approach. Now, organizations have to deliver new, high-quality software to customers as soon as possible to remain atop the competitive market.

    In this new approach, organizations need to focus on two prime factors: agility and adaptability. While the former defines the ability to code and recode, the other marks DevOps’ ability to include changes along the SDLC.

    Companies are constantly working on getting these qualities into their workflows to promote DevOps concepts of continuous delivery, continuous integration and eliminating bugs to allow users to enjoy a hassle-free software experience.

    DevOps Over the Past 18 Months

    According to Gartner, about two-thirds of organizations were using DevOps to support the certain business process by the end of 2017, which Forrester termed as the “Year of DevOps.”

    Forrester awarded the term to DevOps owing to increasing clients’ questions and discussions around, “How do I implement DevOps at scale?” rather than just, “What is DevOps?”

    Going by DevOps’ potential to support modern business processes, more and more companies are adopting DevOps, leading to a steady rise in the trend.

    As DevOps continues to be successful within and across the organization, the IT industry will continue to see what it looks like to be “DevOps successful.”

    The Emergence of DevSecOps

    While DevOps continued to change the software industry, however, security within the process still remained an unanswered question.

    In 2012, Gartner analysts Neil MacDonald and Cameron Haight introduced the concept of DevSecOps. This meant security’s introduction right from the beginning and continuing throughout SDLC.

    Security’s entry into the model made DevOps a more alluring practice. Six years out, DevSecOps has been a much-discussed concept across seminars, conferences and written content.

    But it is still a question whether organizations are implementing DevSecOps considering the time constraint, the need for accelerated delivery and the understanding of security’s importance in the cycle.

    For many organizations, security is still not a part of their production cycle. But, it is essential for any organization to consider security if they want to remain in the competitive market and importantly at the top.

    Negligence in this area may cost them bankruptcy or even elimination from the industry. According to the “Global State of Digital Trust Survey and Index,” one-third of British customers stopped using an organization’s services owing to a security breach.

    10 Years of DevOps: What’s Next?

    The emergence of AI and automation have made some waves in the IT industry, and DevOps is no exception.

    DevOps holds high predictions for the future. In the years to come, AI and automation will change the software development process. Apps will be self-written and coded. Tasks such as integration, security, testing and deploying will be dictated by AI, data and machine learning in the all-new AIOps.

    A data-driven approach will have software dynamically adapt its behavior on the basis of what is in its environment, much like other AI-driven technology including Apple’s Siri and Alexa.

    Moreover, this significant shift from programmed to non-programmed apps will call for improved security and functionality management and allow developers to focus on value-addition activity.

    DevOps is evolving. How about your organization?

    — Veritis

  • How DevSecOps Can Help Avoid Catastrophic Breaches

    How DevSecOps Can Help Avoid Catastrophic Breaches

    Earlier this year, Tesla’s Cloud was hijacked and used to mine cryptocurrency, exploiting a vulnerability in the company’s Kubernetes cluster. A mountain of FedEx data was recently exposed, affecting 119,000 individuals. The Equifax breach garnered international attention after an estimated 145.5 million Americans were jeopardized. In other news, we’ve reported on how the Vine Docker registry fiasco was hacked, leaving an embarrassing PR trail in its wake.

    What is common across all these scenarios? They arguably could have been avoided with basic safeguards underpinned by a healthy dose of DecSecOps. In hindsight, let’s see what specific strategies developers can adopt to avoid such horrendous leaks.

    Basic DevSecOps Can Prevent Vulnerabilities

    DevSecOps is described as placing security at the forefront of every action. Within the world of cloud tooling, this means instilling protection at every point of the build life cycle. You would think this equates to advanced container and orchestration security, powerful access management and establishing hardened oversight for internal applications, right? Actually, many modern exploits simply involve a lack of basic password protection.

    Case Study No. 1: Tesla Cloud Cryptojacking

    In mid–2018, Tesla’s Amazon servers were hijacked by malware and used to mine cryptocurrency for rogue agents. Similar cryptojacking has transpired at Gemalto, Aviva and others. According to RedLock’s report, the hackers infiltrated the Kubernetes console, which was not password-protected. In one pod, they found access credentials to Tesla’s AWS environment. The breach exposed sensitive telemetry data, though was limited to “internally used engineering test cars only,” according to Tesla.

    How to Prevent

    Password protection for Kubernetes administrative consoles is a readily apparent lesson here. All cloud accounts, even if used internally, must be better-equipped and access credentials sequestered. Surely, instilling a “security by design” mentality can help limit the number of such consoles left accessible.

    Case Study No. 2: Vine Registry Breach

    In 2016 a white hacker by the alias avicoder was able to infiltrate Vine, the video-sharing social network. The hacker used tools to discover subdomains and behind one—https://docker.vineapp.com—found an open Docker private registry that housed the Vine source code. Such a vulnerability could have been used to collect user details or inject malware for malicious purposes. Thankfully, no users were compromised, and the Twitter Bug bounty program awarded avicoder a handsome sum for the discovery.

    How to Prevent

    In this particular incident, there was nothing at fault with Docker or Docker containers. The service (which was meant to be private) was simply left public and devoid of access controls. The lesson is that URLs on the world wide web are just that—publicly exposed to the world. It does not matter if no documentation links to them; such subdomains are easy to discover with crawlers and will be exploited if found.

    Case Study No. 3: Equifax Exploits

    In the Equifax breach, millions of customer records were stolen. The Equifax report notes that hackers exploited a “website application vulnerability.” Further reports detailed that the 2017 breach utilized flaws in Apache Struts, likely through a zero-day exploit, according to the Apache Struts Project Management Committee.

    How to Prevent

    Some note that the deserialization of untrusted data inherent in Apache Struts applications leaves some major vulnerabilities and potentially malicious code execution.

    Containers, due to their short lifespan, are not persistent, therefore making consistent hacking harder. They can also isolate functionalities so as to decrease attack vector. Regarding the Equifax breach, Aquasec speculates that container usage could have lessened the impact:

    “… it’s likely that such a breach would have been more difficult with containers, and that if successful, it would have been less persistent, not as widespread, and mitigated sooner.”

    Some view containers as a formidable force if armed with behavioral analysis and firewalls for network connections. Still, others note flaws in assuming containers are safer.

    Regardless of architecture, In DevSecOps, security for customer data isn’t drafted into the background; rather, it is imbued into the entire tooling process and holds equal footing within each build.

    Instilling a DevSecOps Culture

    Instead of a build-now, secure-later attitude, DevSecOps seeks to elevate security into every decision. With each build, security is melded with greater forethought regarding the repercussions for how data is treated.

    Admin consoles for Kubernetes clusters must be armed and password-protected, and the same goes for private GitHub repositories or Docker registries. Any traversal of data through cloud SaaS services must have authorization in place. A most obvious lesson is to password protect those “hidden” registries with discoverable URLs.

    Bounty programs for white hackers continue to lead to successful bug detection. At least for these institutions, there is less evidence of compromised user data and remediation is quickened.

    Much of this can be distilled by due diligence and through breeding a culture of security. While that’s a little fluffy for some, it apparently still requires repeating for institutions that continually de-privilege these basic safeguards.

    — Bill Doerrfeld

  • American Express Uses DevOps for Better Breach Response

    American Express Uses DevOps for Better Breach Response

    It was 6 a.m. Saturday when the phone rang and awoke the American Express CIO. When the phone rings at 6 a.m., it is never good news. Never. This instance was no different: In this case, a third-party supplier just suffered a breach, and that breach would impact American Express cardholders.

    Immediately following that call, the CIO activated the company’s Cyber Crisis Response team. The job of the Cyber Crisis Response team is to help identify impacted cardmembers and prepare to reach out and assist any who have questions or need help regarding the breach.

    The need to respond swiftly to customers and provide them with the correct information is essential today, not only to stay on the right side of regulators, but to also help customers avoid fraudulent transactions and identity theft. When it comes to such effective data breach incident response, rarely does DevOps come into play—but the experience American Express shared at the most recent DevOps Enterprise Summit revealed just how better DevOps organizations can be when it comes to effective breach response.

    Within hours of the initial call between the CIO and the Cyber Crisis Response team, the primary breach response was divvied into three teams. The first team focused on what data breach bridge teams typically focus on: how to identify impacted customers. The second team was comprised of business and product owners, as well as customer care staff, whose objective was to take the findings from the response investigation and communicate them to American Express customers.

    The third team consisted of DBAs and system specialists, who understood all the systems, as well as enterprise architects, who might be able to quickly solve any technical challenges that arose.

    Getting to the Right Breach Information

    By 3 p.m. that Saturday, the first team determined that it could pull together all of the information it needed to identify affected cardholders. That was the good news. The bad news was that there would be tens of millions of production records that would need to be evaluated to make the final determination.

    If the team was to pull those tens of millions of records in production, the demand would start to slow those production systems. “How do we pull these records out of production without impacting our availability? That was the challenge,” said Aimee Cardwell, vice president of Engineering, Consumer Product Development at American Express.

    As the day moved forward and the teams worked to find a way to access those production records without impacting availability, one of the engineers on the team pitched an idea that would have been scoffed at in most organizations: Why don’t we clone production? But the novel idea wasn’t immediately rejected. The team on the call began to weigh the pros and cons of cloning production, and concluded that it could, in fact, clone production quickly, and all of the impacted card data could be aggregated without negatively impacting American Express’s servers and associated availability.

    After the team successfully cloned the production systems, it worked all night to identify the impacted card members, which required cross-referencing the cloned production system with other data stores, Cardwell explained.

    “What was really important here was the comfort level everyone had when it came to bringing up an idea that was really off the wall. And, in the end, they came together to make it happen because together the various teams had a string understanding of the people, technologies, and processes in place necessary to succeed,” said Chad Avery, director, DevOps implementation at American Express.

    It was now 6 a.m. Sunday and the teams had worked nonstop since the CIO’s phone rang Saturday morning. The teams managed to evaluate the relevant data from various systems and collected a list of potentially impacted card members. And after a careful analysis, they were able to determine who had been affected by the breach and who hadn’t.

    What made this success possible? Both Avery and Cardwell believe that it was the integration of business, product and technical teams. “The fact that we had business, product and technical teams working together in this incident was a huge win for us,” Cardwell said.

    Avery contended the ability for an organization to successfully integrate its technical, business and product teams makes the difference. If those teams hadn’t been working together from the beginning, he said, they may not have been able to find a solution and, if they had, it would have taken much more time to do so.

    Disclosure Day, and a Final Big Win

    It’s a good thing it worked out quickly. By Monday morning, the breached third-party was ready to go public with its data breach disclosure. That announcement would immediately impact not just American Express cardmembers, but those of most other credit card providers as well. Many customers would have pressing questions about whether their accounts were impacted, and, if so, what to do.

    Customer service lines had to be ready.

    That proved to be the final challenge that needed to be solved: How would customer service representatives know which customers calling were affected? Tight team integration and collaboration proved to be a deciding factor once again.

    The engineers on the product team had spent the previous 24 hours building a tool that they shared with American Express customer care professionals. When individuals called the customer care team, the tool could alert the team whether a customer was in the impacted group.

    The tool also came equipped with a data collection feature that would enable everyone to learn from the event and even prepare better for next time. “We want everyone to get better. We want ourselves to get better,” Cardwell said. “We want to know what percentage of customers who called were impacted, and what percentage of customers that called were not impacted. We wanted to know our response time on those customers. We wanted to know any other data that would help us to do better next time.”

    And it will have been that dedication to collaboration among technical teams, business units, developers, operations and security teams that enabled them to quickly and successfully manage a third-party data breach and prepare themselves to do even better next time. “They all were able to develop a solution quickly. They were able to identify the customer impact, design a solution, build it, test it and get it deployed. And, oh by the way, the teams did it in a weekend,” Avery said.

    — George V. Hulme

  • DevOps Chat: The State of Software Security with Veracode’s Chris Eng

    DevOps Chat: The State of Software Security with Veracode’s Chris Eng

    Every year CA Veracode publishes its “State of Software Security” report, chock full of research about how well software is being secured during development and beyond. It’s eye-opening stuff and definitely worth a read for anyone even remotely interested in helping make software more secure.

    In this DevOps Chat, Chris Eng, VP of research at Veracode, recently discussed with me this year’s report findings and some takeaways that are worthy of noting.

    As usual, the streaming audio is immediately below, followed by the transcript of our conversation.

    Transcript

    Alan Shimel: Hey, everyone, it’s Alan Shimel, devopsy.kinsta.cloud, and you’re listening to another DevOps Chat. Joining us on this episode of DevOps Chat is Chris Eng of Veracode. Now I’m proud to say, actually, that I know Chris, actually, before there even was a Veracode. But – so it goes back a while. Chris, I don’t remember your exact title, but don’t you head up research at Veracode?

    Chris Eng: That’s right, yeah. VP of research at Veracode.

    Shimel: Yep. Welcome and good to have you back on, Chris.

    Eng: Good to be here.

    Shimel: Yep. So, Chris, you know, I’m gonna assume, if people wanna know who you are, it’s Chris Eng – E-N-G. You could look him up. He’s a bonafide cybersecurity warrior, with a long track record of accolades and just being a great guy in the community. On top of that, though, I’m gonna assume people know that Veracode, for the last couple years, has been putting out a report called “The State of Software Security.” In fact, this is volume nine, so which assumes there was eight before then, right, Chris?

    Eng: That is correct. [Chuckles] We do this every year and it gets bigger and better every year.

    Shimel: Yep. So volume nine, you know, it’s CA Technologies Veracode now, of course. Some changes. And, Chris, it’s kind of your baby, right? You shepherd this report every year, no?

    Eng: I’ve been involved in it since the first year we did it. I mean, certainly, going back to the early years, there were a lot fewer of us sharing the burden and doing all of the writing ourselves. And now, thankfully, we can spread that out among more people and get more eyes on it. And that’s actually one of the things that we did this year that was kinda different, is, in addition to ourselves, we worked with some data scientists, over at Cyentia Institute, to actually help us dig into some of the things we really wanted to probe, around remediation work and looking at different organization behaviors. And I’ll get into that in a little bit.

    Shimel: Absolutely. Chris, before we dive into maybe the specifics of this year, let’s just take a minute and talk about, look, nine years running, my god, how the world has changed in nine years. Right? I was talking to the CIO of Box, Paul Chapman, earlier today, and we were just talking about how the world’s changed, let’s say, in the last 20 years but even in the last 9 years. I mean, what we can – you know, security’s still security, and, unfortunately, we still have incidents; we still have vulnerable software; we still have buggy and have trouble closing up vulnerabilities. But the game has changed, though, too, has it, Chris? When you look back, what do you think that some of the biggest changes are?

    Eng: Yeah, it’s definitely undergone a huge evolution. When we started Veracode in 2006, application testing primarily was companies doing penetration tests, maybe once a year, against their most critical applications. And, if anything changed between those, you know, in that one year, you kinda only had a point-in-time assessment. You had a lot of applications, a lot of pieces of software, that were just never being tested at all. And, since then, companies have realized that, number one, they need to test more frequently. That’s the only way that they’re able to release software that is going to be secure and also be able to drive the speed that they want to release new features and new products. Right?

    The other, I think, realization is that you can’t just focus on the most critical applications to the business because attackers don’t really care how they get in, right? If they find a weaker way to – an easier way to get in, through some weaker application that’s not maybe core to the business but is still connected to the same infrastructure, well, they’re gonna go after that. So there’s been a realization that it’s important to understand everything that you have and to actually build software testing and security testing into all stages of the life cycle, right? Fix early and it’ll be a lot cheaper than fixing later on.

    And, you know, we were looking at some of our stats the other day – this isn’t in the report, but to do the first million application scan for our customers took 10 years; the second million scans took one year.

    Shimel: Wow.

    Eng: Right? So that huge rate of increase is just showing that, across the board, across all industries, company sizes, organizations are realizing that they have to scale their software security programs to cover all the bases. And they have to do it continuously, in order to not rack up security debt. So I think that would be the biggest thing that’s changed in how we approach the problem.

    Shimel: I agree. I agree. But, Chris, really, our time’s limited. Let’s jump into this year’s report. I’ve had a quick look through it, the executive summary and some of the other stuff. Boil it down for our listeners. What do you think are the three biggest kinda metrics or facts that jump out here, that they should be aware of?

    Eng: Yeah. Let me give you just a quick –

    Shimel: Sure.

    Eng: – review on – so, like I said, we do this every year. And we take about 12 months of data, so this is across all of the assessments that we do for all of our customers, and it’s anonymized down, of course, and then we look at trends and try and see what’s going on out there, in terms of application security. This data set is about 700,000 application assessments during a 12-month period, and the one big thing that we’ve been able to do this time that we weren’t able to do in the past is actually look at how long flaws are sticking around once they’re discovered. Right? How long is it taking organizations to fix these flaws?

    And so, when we detect something, we’re able to kinda track it across multiple assessments, so let’s say we scan an application today, we find a bunch of flaws, we fix some of them and not others, and then we scan it again in a week. We’re able to identify that certain of those flaws, the ones that we detect the second time, are the same as they were the first time. And so that allows us to actually track the life cycle of each individual flaw until it’s fixed, right? And so we’re able to look at “What does that flaw persistence look like?” Kind of a survivability analysis.

    And, if you look overall, just across all the flaws that we look at, the numbers are not great. It takes 21 days on average to close 25 percent of the findings, 121 days to get to the halfway point, and 472 days to get to the 75 percent point. So that’s a long time, right? That’s over a year, so it’s not a very rosy picture overall. And so we broke this down and, you know, once the report is out, people can kind of look at “What does that look like in terms of different severities, different industries? What kind of patterns do we see there?” But I think the most interesting one, and the one that’ll be most interesting to you, is around the effect of DevOps or DevSecOps on fix velocity.

    Now we don’t have an easy way of saying, for any given application, “Is this a DevOps shop or not?” Right? We can’t – we don’t have that information, so we have to figure out whether or not we think they are, and the proxy that we’ve been using for that is scan frequency, right? If you’re doing CI/CD, you’ve got your scanning baked into your build process or you’ve got it, some way, baked into your automation. And so, if you’re doing a nightly build, for example, you’re doing a test every night. And so we looked at scan frequency to identify that – you know, we think that probably these applications that are being scanned 300-plus times a year, those are probably DevOps shops, right? And the ones that are scanning one to three times a year, probably not.

    And so we’ve got this broken down into buckets. If you have this in front of you, you wanna jump over to page 39 – it’s Figure 43 – and this is what I really want to – this is sort of a – our hypothesis was like, “Yeah, I bet DevOps shops fix faster because they’re scanning more frequently and they don’t wanna rack up as much security debt.” And, in fact, it does show that. Right? We see this one line on the report that says, “Applications that are scanned 300 or more times a year are fixing – they’re getting to the halfway point of fixing in five days and they’re closing 75 percent of flaws within seven days.” Right? So that’s a huge amount better than those averages that I just described to you.

    And we have this broken down by bucket, right? So you have the 300-plus scans per year, you have 51 to 299, all the way down to one to three scans per year, where it takes close to four years for them to get to the 75 percent close mark. So I think that’s actually – this is what we hoped to see. This is what we expected to see. But it’s nice to see the data prove out that, in fact, yes, when you scan often, when you scan early and you stay on top of the findings, this is actually a very doable thing, right? To fix flaws in a reasonable time frame.

    Shimel: Absolutely. Two things there, Chris. No. 1, I would think hand in hand with the DevOps is automation.

    Eng: Yeah.

    Shimel: Right? I’m wondering if part of it is that, in DevOps-enabled cultures and organizations, organizational cultures, whatever – companies doing DevOps – are they automating the scan process and that’s why we’re seeing them doing 300-plus scans? Right? And then they’re deploying more and, if their MO, if their standard process is “Nothing gets deployed without first being scanned,” right, that’s where you start racking up 300-plus scans. Right? I mean, when you –

    Eng: Oh, yeah. Yeah, we’re –

    Shimel: – build that into your pipeline, right, you build that into your CD process, those scans get done.

    Eng: Yeah, we’re assuming that these – you know, we’ve gotten – I think, at the maximum point, there’s one app that was scanned over 1,000 times and, certainly, that’s – I mean, I really hope that’s not a human uploading something or pressing –

    [Crosstalk]

    Shimel: Right, _____ – hitting the “Scan” button –

    Eng: – a button every time, right?

    Shimel: – every time. You would hope not, right?

    Eng: Right. Right. I’m assuming that even the ones that are scanned 50 times a year, right, which I assume to be every week, I assume that those are also –

    [Crosstalk]

    Shimel: Are still better than once _____ –

    Eng: – _____ _____ through a build pipeline. Right? And, any time you can – I mean, and that’s been a big focus for us, just in general, is “How do we remove any human effort required to do the testing? How do you make it easier and more transparent and just remove any barriers to doing that?” And so –

    Shimel: No, heck, how do you – you gotta build it into your CI/CD pipeline. I mean, it has to be part of that. The same way you breathe air, right? If you stop breathing, you die; you don’t deploy software that’s not been tested and scanned. And –

    Eng: Right. Right.

    Shimel: You know, it sounds so simple, us talking about it, Chris, but here’s the fact, right? And not everyone listening to this in our audience has been in security as long as you have or even I have. And a lot of people see the reports, Chris, and they say, “Oh, my God, this is year nine,” and I forgot how many years Verizon is doing their report, and, every year, it’s hard to show progress in some things, right? We do show “Hey, if you scan more, you’re gonna have less vulnerabilities go out the door that aren’t fixed and that were at least – they don’t stay – you know, they get fixed faster.”

    Eng: Yeah.

    Shimel: But the fact is we’re still seeing – you look at the executive summary here, Chris, and it’s not a rosy picture, as you – I think those were your own words. And we still read about breaches every day and 30 million records here and 20 million here. You know, the non-security person says, “What are we gonna do? When does this fundamentally change?”

    Eng: Yeah, I mean, things don’t get solved overnight, right? I mean, if companies have only realized a few years ago that they need to kinda get a handle on all the things that they have and start understanding what their risk posture is, and they’ve got 10, 20, 25 years’ worth of software, thousands of applications, it doesn’t happen overnight; it’s baby steps. Certainly, starting any new greenfield projects with a proper SDLC that’s identifying and fixing issues early is gonna prevent them from building up even more security debt.

    But, yeah, a lot of these – the way I explain some of these things – for example, pass rates on OWASP Top 10 not getting a lot better or the prevalence of certain flaw categories, like cross-site scripting and SQL injection, staying relatively constant – is that some of the more mature apps that have been scanned for a while now, those are actually decreasing. If we sliced those separately, we would see the rate going down. But, when we look at it in aggregate, out of those 700,000 applications that are part of this data set, a lot of those are things that are just being scanned for the very first time ever. And so they’re gonna have a higher prevalence of some of these issues, that the more mature applications will have gotten rid of already. Right? And so they kind of balance each other out.

    And we see this every time. In fact, we should probably start to report on some of those things separately – it’s kind of a difficult thing to do – but I think what’s happening is some of the newer applications are bringing those numbers up and that’s why we see the lines staying relatively flat. There’s a long way to go, but we do see – you know, and things like the DevOps remediation case I just described. I mean, we see little points of optimism, where you can see a clear correlation between a certain behavior and a certain outcome. And so we like to focus on those things, and, yeah, it’s gonna take a while for most companies to really start to chip away at the legacy applications and getting those locked down.

    Shimel: Absolutely. Absolutely. Chris, we’re just about out of time. I should – we should put this out there: for people who are interested in really kinda digging in on the report, where can they get it from? Go to Veracode.com?

    Eng: It’ll be on Veracode.com, starting on October 24th, and, yeah, they can download it there. There’s a huge amount in here. We break it down into different flaw categories. We look at different industries against each other. There’s even a little geographic breakdown and then, of course, like I mentioned, a huge amount of data around flaw persistence and how long it takes organizations to fix flaws, which, I mean, ultimately, is what’s important, right? Doesn’t matter how often you scan if you don’t fix stuff. Right? You don’t reduce your risk unless you fix stuff. So there’s a lot of great data on that, that we’ve never been able to report out before, so we’re really happy to get those out there.

    Shimel: Absolutely. Well, Chris, you know what? I know I got a little hard on you about the – and it’s not your fault that the security industry – and it’s not the security industry ’cause, God knows, we wanna fix the issues and the problems that cause these breaches and things; it’s really just the state of software and where security lies as a priority in a lot of these organizations and managing their risk and so forth.

    But, by the same token, Chris, you know what? First of all, thank you, thank you, thank you, for being involved enough to do this report every year, and, whether the metrics paint a rosy picture or a sad picture, nevertheless, it’s a picture that needs to be told, right? We need this kind of insight; we need the metrics to understand where we are and where we need to be. And so great job, once again, in putting this together. You know, I think it’s as important for non-security people to read these reports as it is for security people. Right? To really –

    Eng: Yeah, we hope –

    Shimel: To –

    Eng: Yeah, we definitely hope people will get something out of it and, hopefully, kind of use that to think about how they are building their own programs, right – how they’re remediating, what do their numbers look like – and, hopefully, kind of informs how different organizations think about scaling up their programs. That’s what we’re hoping, anyway.

    Shimel: Good. All right. Chris, let’s call it a wrap on this; it’s a Friday afternoon here. And, again, thanks for your insight on the report. Thanks for managing this report year in, year out and being involved with it and helping. And we’ll see you soon. We’re coming into conference season; I’m sure you’ll be busy presenting and getting more –

    Eng: I’m sure, yeah. I’m sure we’ll be bumping into each other pretty soon.

    Shimel: Absolutely. Chris Eng, VP of research at CA Technologies Veracode, our guest on this episode of DevOps Chat. This is Alan Shimel, everyone. Have a great day.

    — Alan Shimel

  • Security Through ‘Legisurity’: Impacts of SB-327

    Security Through ‘Legisurity’: Impacts of SB-327

    With more devices connecting to the internet every day, security through obscurity should have died a long time ago. Still, high-profile cases of devices with mass-produced default login credentials and easily cracked crypto keys persist. As a result, say hello to security through “legisurity”: California SB-327—“Information privacy: connected devices”—became law Sept. 28. How will this impact DevOps teams?

    What SB-327 Does and Doesn’t Do

    Here’s a quick look at what SB-327 does, starting with its timing and scope. The provisions go into effect Jan. 1, 2020, giving teams a little more than a year to prepare compliant connected device designs. The legislation defines a connected device as one “… capable of connecting to the Internet, directly or indirectly, and that is assigned an Internet Protocol address or Bluetooth address.” Authentication is “… a method of verifying authority of a user, process, or device to access resources in an information system.”

    The operative passage in SB-327 calls for a “reasonable security feature … appropriate to the nature and function of the device.” (Before you get too excited, it explicitly shuts the door for individuals suing manufacturers over non-compliance.) With implementation left open, the legal standard is very clear. Reasonableness is a device with features meeting either of two requirements:

    “(1) The preprogrammed password is unique to each device manufactured,

    “(2) The device contains a security feature that requires a user to generate a new means of authentication before access is granted to the device for the first time.”

    Also written into SB-327 is a list of things it does not do. The law does not extend these requirements to third-party software a user installs, and it does not force retailers or software providers to review compliance of a device. It doesn’t apply to devices sold only to federal agencies or the military, and it yields to HIPAA requirements. It does not permit manufacturers to hinder law enforcement agency access to a device with these required features.

    What Should DevOps Do Now?

    As with most first attempts at legislation, the unwritten implications of SB-327 may be even more interesting. Here are a few things I see changing for DevOps teams securing devices and infrastructure:

    • SB-327 only applies to devices sold or offered for sale in California. However, setting up California-only design efforts, supply chains and customer support would be costly. (For $20,000 cars? Sure. For $200 devices? Not so much.) Practically, I see these becoming de facto requirements nationwide by the 2020 rollout. Companies should start now to be ready.
    • If you’re a device manufacturer, you should add both requirements into your minimum viable product baseline, if you haven’t already. You’ll obviously need a strategy for assigning and tracking default passwords and a crypto key authentication method for setup connection. Customer support procedures and documentation also need to change. Consider using one-time programmable non-volatile memory or PUFs (physically unclonable functions) to establish a unique device identity.
    • On the deployment side, you’ll need added tracking for unique default passwords in your device provisioning database. Your customer support teams will need device-specific information at their fingertips to get new or factory-reset devices back online quickly. Crypto key authentication methods will be a more important criterion for device evaluation and selection.
    • If you don’t have a good UX person and an experienced pen tester on your team, hire or contract those roles right now. Simplicity with security is always going to be the right answer.

    More Laws Ahead?

    Will the debut of “legisurity” satisfy lawmakers and stem connected device breaches? My guess is this is only the beginning; SB-327 does nothing about requiring stronger passwords or thwarting more sophisticated protocol-based cyberattacks. The main point of SB-327 doesn’t seem about securing a single device, although it should help; rather, it’s trying to prevent rapid compromise of an entire population of that device. (Pro tip: whatever you do, don’t use a device’s MAC address in a simplistic algorithm, or you’ll end up like this sad, not-so-smart lock story. Cracking one device cracked them all.)

    For the more curious, here’s the entire text of SB-327, and some other analysis on legislative proposals at the federal level. What are your thoughts on SB-327, how you see it affecting DevOps, and if legislation is the right path for securing connected devices?

    — Don Dingee

  • DevOps Chat: disrupt:Ops Brings Security Closer to DevOps

    DevOps Chat: disrupt:Ops Brings Security Closer to DevOps

    Shifting security left during the software development process can be tough for companies because of the sheer number of operational things that go along with implementing security that must occur somehow in the realm of DevOps. It can be easy to lose control over their security processes without some sort of guardrails in place to help.

    A brand new company is helping do just that. I’m excited to introduce disrupt:Ops, which bills itself as a guardrails platform to help companies include and enforce best practices across security and operations, bringing security into the DevOps process at the beginning. My friend Mike Rothman, disrupt:Ops co-founder and president, joins me in this DevOps Chat to discuss the vision of his company and how it hopes to help make security truly a part of DevOps.

    You can read more about disrupt:Ops and their guardrail approach to cloud security and devsecops from another co-founder of the company, my friend Rich Mogull at https://devopsy.kinsta.cloud/building-great-cloud-security-guardrails

    As usual, the streaming audio is immediately below, followed by the transcript of our conversation.

    Audio

    Transcript

    Alan Shimel: Hey everyone, it’s Alan Shimel, devopsy.kinsta.cloud, and you’re listening to another DevOps Chat. Breaking some big news today on DevOps Chat, the launch of a labor of love of some friends of mine. I’m happy to be joined by Mike Rothman, cofounder and president of disrupt:Ops. Mike, welcome.

    Mike Rothman: Alan, it’s great to be here, thanks. It’s always fun to catch up with old friends.

    Shimel: Yep, and old friends it is, Mike. We should let our audience know – maybe we shouldn’t let them know. But we are old friends. We go back a long time now, back to the beginning of blogging and security –

    Rothman: That’s right.

    Shimel: It’s got to be 2004, 2003 or something like that.

    Rothman: Yeah, that’s right, that’s right. When I had dark hair. Actually I didn’t have dark hair back then, but you know, it’s –

    Shimel: I had hair.

    Rothman: You had hair. That’s right.

    Shimel: I had hair. But you know what, Mike? I said it was a labor of love and everything else. Just in the way of background, why don’t you share with our audience your personal sort of journey in tech and, you know, work history.

    Rothman: Well sure. I started out, God, in what feels like 100 years ago, as a mainframe programmer. Ended up doing some local area networking stuff. Kind of found my way to network security. Went into research for quite a while. Did a couple of companies and then went back to research and for the last 10 or 11 years at this point I’ve been partnered up with Rich Mogull and Adrian Lane at Securosis, doing a bunch of kind of information, security-centric research. And actually about a year and a half ago we spun out some technology that we had prototyped within Securosis in cloud management and cloud automation into our new company disrupt:Ops, and have been just making a lot of progress over the last year with that. And that’s what we’re announcing this week.

    Shimel: Yep. So Mike, you’re being humble. When you say you did research, you were an analyst at Giga, were you not?

    Rothman: META Group, yes. META Group, exactly.

    Shimel: META Group. And then you were also – you had executive positions in several cyber – we didn’t call it cyber – infosec companies back then. 

    Rothman: It was. Yeah, you bet. So companies like TrueSecure and CypherTrust, you know, back early in the e-mail security days. I did a network monitoring company called EIQ. So yeah, I’ve been there, done that. Have the road rash and kind of the screw-ups under my belt to prove it. But you know, it’s interesting in terms of just how long and how small an industry security is for being actually a really big thing nowadays. 

    Shimel: Being so big, yeah. It really is. And Mike, again, for many people in our audience who come from DevOps or even from security – but they’re worldwide and they not be familiar with Securosis. You know, Rich Mogull comes over from Gartner. Adrian’s been CTO at major companies. You know, you guys are probably the leading, let’s call it independent security analyst research firm – you said 10 years-plus. Certainly in my time you guys laid the fundamental kind of – a lot of the fundamental backbone of what Cloud Security Alliance teaches and so forth. 

    Rothman: Yeah, that’s exactly right. So it was interesting. So I guess it was probably eight and a half years ago we partnered up with the Cloud Security Alliance and helped them build their curriculum for their CCSK certification. And that got us to be much deeper than your typical analyst in terms of having to play around with cloud, having to understand automation, having to build a number of the kind of scripts and – I’ll call it the inherent messiness of building out a stack in the cloud and really making it operational, because we were eating our own dog food. Our website, a lot of the stuff that we had prototyped was really demonstrative of the challenges that organizations face as they move a bunch of their stuff into the cloud, and really as they embrace a DevOps mindset of trying to be agile, trying to be innovative, trying to move towards more of a continuous type of deployment model. We really came across a number of the challenges these organizations have in terms of just really maintaining control, right? Once you have multiple DevOps teams doing things in multiple kind of accounts, you start to run into challenges relative to how you enforce best practices.

    And that’s really one of the challenges that security people have with this whole DevOps revolution, which is, “Hey, this sounds great, but we’ve got a playbook, a set of practices and policies that we really require all of our environments to adhere to in order to protect that data.” And obviously the faster you move, the more agile and continuous you try to operate, the harder it is to enforce those best practices.

    Shimel: More about that in a second. You touched on two things there that we’re going to dig into. Let me just finish laying my groundwork. Mike, besides you, Rich and Adrian, the Securosis gang, there are at least two other key people on the founding team of disrupt:Ops who also happen to be friends of ours for years –

    Rothman: Yeah, that’s right.

    Shimel: Give them their due. Why don’t you give them a shout out?

    Rothman: You bet. So back when we spun the technology out of Securosis, we brought on two guys, very experienced guys in the security space. Our CEO is Jody Brazil. So Jody was a cofounder and long-time CEO of a company called FireMon, which I know you’ve done work with for a long time –

    Shimel: Sure.

    Rothman: So Jody kind of is really the driving force behind a lot of what we’re doing at disrupt:Ops. And our CTO is a guy named Brandy Peterson, another longtime security professional. He was actually the CTO of FishNet Security for 15 or 16 years.

    Shimel: As was Jody before Brandy, right?

    Rothman: Yeah. I met both of them around the same time.

    Shimel: Yes, when I first met Jody, he was CTO and FishNet, and he said, “Who is this Jewish guy with the New York accent?” I went out to visit him in Kansas City to tell him why you shouldn’t use Qualys, and he should use StillSecure, and he made fun of my accent. [Laughs]. Good times. Anyway. But Mike, let’s turn back to a couple of things you were saying about kind of the genesis driving disrupt:Ops. You touched on one thing; you said, “multiple DevOps teams.” So this is something we see, and I call it almost second-generation DevOps. You know, when a company or an organization is first embarking on their DevOps journey and they do sort of a trail balloon, and they pick a project and they have a DevOps team – you know, and they may not call it DevOps. They  have a team that’s practicing DevOps sort of principles on this one particular project. And oftentimes, wow, great success, because you are – you know, it’s an isolated environment, right? In some ways you’re not in the quote/unquote real world.

    And then you say, “This is great. Let’s take it out. Let’s expand that beachhead. Let’s expand the bubble.” And all of a sudden – you know, larger organizations; small teams or smaller organizations not so much, but larger organizations – now you start having two, three, four, six different projects that are being run with a DevOps mindset and you have multiple instances of Jenkins and you have three different – this team’s using Puppet; that one’s using Chef. And serverless. And all of the complexities that we see in today’s modern software factory. And it’s chaos, right? And then at some point the CIO or VP or director of IT says, “Oh my God, this DevOps thing isn’t working so good. It’s all over the place.” Let alone the security guy’s screaming, you know, right out of the Phoenix Project, “No, no, no!”

    Rothman: That’s right.

    Shimel: And that’s a problem that we see with a lot of second-gen, if you will, DevOps installs, where it goes beyond the initial team to several teams.

    Rothman: That’s right.

    Shimel: That’s what you guys see.

    Rothman: We do. And it’s interesting, because what you have are really kind of two paths at that point, in terms of how these teams start to behave when starting to be put under scrutiny in terms of, kind of again, their ability to enforce operational and security best practices. You know, one group kind of buries their head in the sand and continues to move forward and really don’t do much, right? They’re like, “Oh, it’s okay,” and you know, they’ll maybe run a vulnerability scan to kind of prove that they’re sort of within policy, but they really don’t have an operational environment that is built to work in concert with the rest of their factory. I love that term, right? That really does encompass a lot of how software is being built today. And then the other part ends up being folks starting to build their own scripts, right? They take maybe an open source-looking thing, they start to build a bunch of their own scripts and code around it, and then what they find is they’ve got a suitcase full of these scripts that they end up having to maintain, upgrade. They don’t have any means of reporting or any scheduling for how they enforce and run a lot of these scripts.

    It’s really – you know, we call it automation, but it’s manual automation, which I get is a total oxymoron. But that’s really how it is. You know, you have a bunch of folks, they’re building these scripts out, they’re executing them manually or through very simple triggers, without any means of maintaining, upgrading or scheduling or reporting on what these really key aspects of their operational environment are doing. And it becomes, again, a big problem, where you have – it’s unwieldy because of just the sheer number of things that are happening in a modern DevOps environment with multiple teams. Then you compound it with a number of very tactical scripts that are built to surround the environment, if you’re doing anything. And then at some point somebody says, “Hold on a sec, folks. This is just not where we want to be. We really need to think about it more from an operational excellence perspective and make sure we’ve got an environment that we can enforce our best practices for security and operations.”

    And that’s kind of when they get the sense that, “Well, I’m either going to have to make a significant investment in this and become like” – Rich calls them the unicorns of cloud land, right? You know, the Netflix and the Capital One and, you know, the folks who have whole teams of environments that are building tools to do this. Or they have to start looking for some kind of commercial option, because there really is no option, but you’ve got to do something. You can build it yourself or you can look to buy it. And candidly, until the last handful of months, there really wasn’t a realistic commercial offering to start implementing some of these capabilities.

    Shimel: No, no. I think the bigger – you know, a lot of what you’re saying certainly obviously rings true. The other fundamental piece of this is, even if you are using one of these homegrown, cockamamie, you know Rube Goldberg kind of things –

    Rothman: Yeah.

    Shimel: – is it the security guys who are? You know what? Shame on me. It’s not the guys. Is it the security team who’s doing this, people? Is it the DevOps cross-functional team? Is it a little bit of both?

    Rothman: Yeah, yeah.

    Shimel: You know, how many of – look, here’s my basic, fundamental belief. You are not going to find a developer who raises his hand who says, “I want to write shitty code. I can’t wait to put something insecure out there.” No one says that. No one says that, right? They’ll tell you they’ve got time pressure, there’s pressure to release, pressure to release, they wish they could do more on security. And some of them, the enlightened ones, as you said, they’ll even try. They’ll run a vulnerability scan, because there are certainly open source kind of tools out there for them to mess with.

    The problem is, they go to a lot of the security guys, and as you alluded to before, security guys say, “Well we have our rules and procedures here. This is a no.” Right? And they don’t want to hear no. They’ve got to go.

    Rothman: You know, what’s interesting, Alan – I want to hit on a couple things that you alluded to there, because they’re important, right? The first is kind of where you start to see the agitation for better operational and security practices. And it’s not that it’s not the security folks. Obviously they’re always going to push for that. But it really is – you mentioned it, right? The DevOps teams that we’ve worked with, they’re very cognizant, they’re very serious stewards of kind of the corporate data, and they want to do the right thing. But in a lot of cases they don’t know what to do, right? So you have a whole bunch of folks who don’t know what they don’t know and they don’t want to go to the internal security team because of a lot of those things you were just alluding to. They’d say, “No, we can’t do that. We have all of these rules.” And in a lot of cases the security team has not updated their set of conditions, their set of policies – 

    Shimel: Exactly.

    Rothman: – to work in a DevOps type of environment, so these folks really struggle. They talk to – again, on the consulting side of our house at disrupt:Ops, we do a lot of work with those teams in terms of helping them understand what cloud native architecture looks like and how to integrate cloud security into their environment, and really just to help them build a program in order to do that. So then you do that with one DevOps team, then you end up talking to another DevOps team and maybe the third DevOps team, and finally IT, at some point, they’re like, “Um, what are you telling our folks? They seem to like what you’re doing and they seem to be doing things in a more secure way. So can you help us?” And then IT starts to get religion. The security team starts to get a little bit of religion about what they have to evolve to. And it really is kind of that interesting transition where it has started kind of out in the DevOps community with, again, those enlightened ones going, “I don’t know what I don’t know. Let me get some help,” and then it kind of back-ending into security. That’s not always the case, but probably three quarters of the time, that tends to be the transition that we see.

    And the other thing I want to hit is just about velocity, right? JA very forward-thinking DevOps leader that we’re working with, he said it the best. And he’s just like, “I’ve got 20 DevOps teams. I cannot slow these people down. They are doing stuff; they are making progress. The last thing I can do is put a bunch of onerous security requirements on what it is that they’re doing. They’ll ignore me, they’ll go around me and that’s counterproductive. I can’t do that. What I really need is to be able to build and enforce a set of guardrails around their environment.” Right? That term, “guardrails?” It just really kind of opened up and was a light bulb in our head when we were having this conversation. It’s not about stopping them from doing things. If anything, it’s about allowing them to go faster, yet making sure they don’t drive the car off the road, right?

    And that’s really the metaphor that resonates relative to what we’re doing at D:Ops, and really just kind of where the environment is going, in that we have to be able to move faster. Slowing things down is not an option, yet you want to make sure that you are not putting corporate data at risk. You are not opening up significant new attack services just – because people are moving so quickly. Sometimes they make mistakes. Sometimes automation isn’t done in the perfect way. Sometimes humans get involved and certainly are fallible from that standpoint. But with a lot of the inherent, underlying automation orchestration that’s possible within the cloud and DevOps and continuous, you have options to enforce these best practices in a way that we couldn’t do in our traditional data center with our traditional waterfall types of development and operational motions.

    Shimel: Absolutely. And you know, Mike – I mean, I’ve spoken about this a lot at various conferences and stuff. A lot of that has to do – you know, we use this term “shift left” in security and all that. A lot of it does have to do with getting to security earlier, faster. Yeah, we’re going to have to automate it. Yes, we can’t be the roadblock. But we can be the appendix after – when I say, “we,” I mean security. We can be the appendix after the delivery or after the deployment.

    Rothman: I’ll raise you. I see your point and I’ll raise you one, right? I’m of the opinion that security isn’t a thing. Right? Security should be within everything.

    Shimel: Yeah.

    Rothman: And that’s really the mentality that we’ve been trying to push on the industry for a long time.

    Shimel: Amen.

    Rothman: And it’s one of these things where we really can’t afford to wait and be in a position where we’re trying to retrofit poorly architected systems that are moving faster than we’ve ever seen before and then say, “Oh hey, we’ve got to change it because of security,” or because of any other operational gap that we see there. Right? What we have to do is be building in a way that allows us to use architecture as security control, right? Isolation, segregation, multi-factor authentical, identity management – all of these things that are fundamental building-blocks of what we do in the cloud and DevOps can be utilized to build in a much more secure fashion. And it’s really about those kinds of practices that are going to make the biggest difference relative to what it is that we see and what it is that we deploy over the next couple of years.

    Again, like I said, I don’t think security is a thing five to seven years from now. Because if it’s not embedded within everything that we deploy, we screwed up.

    Shimel: Yep. You know, I heard – because I never come up with original ideas. I just listen. But I heard something awhile back – “Security needs to become synonymous with quality.” And we’ve had QA as a concept in software technology forever or for a long time. Security needs to be just built in, like as part of the quality control. Right? The same way shit or stuff gets tested, whether it be hardware or software or whatever. Security needs to be part of that quality. 

    Rothman: You bet. To use a software term? Right? Why is a functionality defect different than a security defect?

    Shimel: Exactly.

    Rothman: Right? And they should all be within the same thing. And again, that’s a lot of when you start talking about immutable, when you start talking about kind of building security testing into your pipeline, which again, is stuff I know that you guys at devopsy.kinsta.cloud write about quite a bit. Again, it’s really not any different than you managing and making sure that you hit your functionality kind of specs from the standpoint of just building code, right? Part of that factory. So yeah, the code has to work, but it also can and should be secure, and there really aren’t different operational motions to make that happen. And that’s what’s so exciting. For a guy who spent 25 years trying to retrofit and get people to think about security as part of their core structure and to treat it as really a business function, it’s really enlightening to be in a situation where you actually can implement a lot of those functions within the core processes that you use to develop the technology.

    And again, are we there yet? Of course not. Is there a lot of education to be done? Without a doubt. But we at least have the option now, right? That’s the aspirational goal to what it is that we all do every day, which is to make sure that we’re pushing for the right types of decisions to be made as early in the process as we can, because we all know how much it costs to fix it later.

    Shimel: Absolutely. Mike, I’ve got to tell you, listening to this conversation, I’m thrown back to, “NAC: is it a feature or stand-alone?”

    Rothman: You’re dating both of us, man. Stop that out. Cut that out.

    Shimel: It’s such a pleasure to talk a little theory here with you. But let’s turn for a second to real stuff. Not that what we’re talking about isn’t real. But I wanted to give our audience a little bit of a taste – so disrupt:Ops – great people, great big problem to tackle out there. Give us some specifics. What are you guys doing?

    Rothman: Yeah, so what we’re doing is we’re basically building a cloud management platform to implement a lot of these ideas that we’ve been talking about for the last couple of minutes, Alan. Initially it’s a guardrails platform. So what we have is the ability to, in an AWS environment, enforce a set of best practices across security, operational and also cost management. So one of those things we don’t talk about too much is that developers don’t really like to shut their stuff down because, “No, it’s going to mess with something. I don’t have to. The CFO bitches but we don’t really care.” What we can do is also implement best practices in order for folks to clean up their room, right? We try to do that with our kids all of the time, too. You know, so we try to do that with our developers as well.

    So really implementing a platform for security, operational and cost management guardrails, to again, immediately not just find the issues – there are a lot of tools, as you said. A lot of vulnerability scanners can kind of look at your cloud environment and tell you you have problems. The issue with that is that most operations people are overwhelmed with the number of other things they have to fix every day. Adding more stuff to their list doesn’t really help them. So disrupt:Ops is really focused not just on finding the issues, but also fixing them and enforcing those best practices. Let me give you an example of that. So let’s just say some person decides they want to open up a server to the internet because it’s just easier for them to do that. Well obviously that would be a pretty significant security problem. So as opposed to just taking that server offline, what you can do is you can basically change the security group to make sure that it’s only accessible via a known, good IP address range within your corporation. That’s a good example of a guardrail. Or making sure that you’ve got MFA turned on for every account that gets spun up by your magical Terraform environment that may not build that out as a core security requirement.

    So again, there are 50 to 60 of these guardrails, in effect, that you can implement in your environment, to again, make sure that you are not just alerting when you get out of policy, but actually fixing the issues, in a lot of cases before you even know they’re issues, so that you’re not sitting there trying to pick up the pieces after half of your data is in Chechnya.

    Shimel: Exactly. Or Romania.

    Rothman: Or Romania. It’s somewhere.

    Shimel: Mike, how is the offering packaged? Is it a SaaS kind of offering? 

    Rothman: Yeah, it is. What we found is since – you know, remember the days where people were like, “Oh, I don’t know about this cloud thing or SaaS. I’m not sure I’m comfortable with that”? Now it’s like, “Wait, you want me to install software in my environment? What? What’s wrong with you?”

    Shimel: That would be kind of oxymoronic for a DevOps tool, but –

    Rothman: That’s right. That’s right. So we deliver as a SaaS service. It’s actually a very cool kind of one-click provisioning process where, when you’re logged into your AWS account, you run basically a cloud formation script that really kind of gives us cross-account privileges to your environment. And again, we’re security guys, right? So at the end of the day, we are basically not even just assuming but enforcing least privilege. So we only have read-only access. When we need to make changes, we grant the privilege as we need it, then we revoke it. Because again, what we’re trying to do is make sure we enforce operational best practices; not add more attack service. So again, we’ve built the environment to be secure by design. We use all of our own dog food, to make sure that we are adhering to kind of common best practices. And again, we’re all 25-plus years security folks, so we do take that pretty seriously.

    Shimel: Excellent. Mike, for people who maybe want to download – not download, excuse me – sign up for it or check it out more, where can they get more information?

    Rothman: Yeah, disruptops.com. So by the time you hear this little interview, the site will be live. There’s a place you can sign up. You know, first we start with a demo and show you what it’s about and then you can sign up for early access. And again, we’re kind of expanding towards the end of this month, in October. We’re going to add a bunch more folks to the system with the goal of kind of being ready to actually sell the product by the end of the year.

    Shimel: Fantastic, man. Well, I don’t have to tell you, Mike. I wish you, Rich, Adrian, Jody, Brandy and the rest of the team nothing but success. We’ll be checking in, I have a feeling, pretty often here, whether it be through Security Boulevard, devopsy.kinsta.cloud, Container Journal or what have you. And you know, fantastic. It’s good to have you back in the game.

    Rothman: Yeah, no, I’m excited. Again, really it’s not just a big promise and not just a huge opportunity. You know, and what Rich, Adrian and I, and now Jody and Brandy have always done is – you know, we’re trying to do the right thing to evolve the practice. Right? And that’s what it’s about. It’s a big world and there’s a lot of opportunity. There’ll be a lot of folks talking about guardrails over the next year. And that’s great, right? Because what we want to do is get to a position where we can move faster but move more securely, and I a lot of cases, in the world that we grew up in, those were counter indicators. Right? You couldn’t move fast and be secure. You couldn’t do operational excellence and make sure that you’ve got a lot of these things enforced. And now you can, and that’s what’s really exciting to us. Again, we’re just excited to partner with you, as always, and again, just go out and solve some bring problems for a lot of organizations.

    Shimel: Fantastic. All right. Mike Rothman, president and co-founder of disrupt:Ops, a hot new entrant in the DevSecOps space, with a great pedigree and a real reason for being our guest today on DevOps Chat. I hope you’ve enjoyed this. This is Alan Shimel for devopsy.kinsta.cloud. Until next time, everyone, have a great day. Bye-bye.

    — Alan Shimel

  • DevSecOps and Development: Making the World Safer One Application at a Time

    DevSecOps and Development: Making the World Safer One Application at a Time

    If you have been around software development much at all in the past five or 10 years, then you’ve certainly heard of DevOps and know it’s here to stay. And it is very likely that you have also heard the term “DevSecOps” even more recently. With today’s extensive proliferation of application capabilities being delivered through services—and all the inherent complexity that comes with the delivery systems for them—it’s no wonder that cybersecurity has moved to the forefront for many companies. In fact, according to research from Stratistics MRC, the current global application security market is growing at a CAGR of 23.4 percent and is expected to reach more than $10 billon by 2023.

    But it is one thing for there to be an interest in cybersecurity; it’s another thing entirely to implement it well—and that is where the conversation gets more interesting. Today, the name of the game in application security is all about bringing the role of the developer into the fold and equipping developers to code, fix and maintain security as part of the software development process. In an article on DevSecOpsDays.com, DevOps thought leader and author Jon Willis offers the following comments:

    Fast forward to February 2017. I was speaking at the RSA Conference in San Francisco and I started seeing for the first time people talk about security as an interesting new way of integrating it into DevOps. They were unapologetically calling it DevSecOps. They were talking about this in a way that included security as a systems (end to end) approach to the traditional DevOps pipeline.

    This was not the first time I had heard people talk about security and DevOps.

    The thing that most caught my eye was that this approach was clearly putting the ownership of security on the developer. I immediately thought back to Werner Vogels’, the CTO of Amazon, statement of “You Build It, You Run It”, back in 2006. However, what I was hearing this time was “You Build It, You Secure It”. This thing they were calling DevSecOps was describing a holistic and systems approach to security as part of the software delivery pipeline.

    This notion of developer empowerment being a critical aspect to success was something I heard again and again at the recent 2018 Black Hat USA Conference. You can read more about my experience at Black Hat in this DZone article. But how do you effectively bring developers into the security fold? That is what we will explore in this multi-part series.

    In this first article, we will focus on what I (and many others) believe is the single most important factor for long-term success: the organizational culture. And by culture I am talking about the day-to-day environment of the organization the developer is a part of, and how they are (or are not) allowed to do their work. A successful DevOps culture is one that promotes collaboration to foster innovation and speed delivery, and the following two sections of this article depict perhaps the two most critical components for a strong DevOps culture: leadership and trust.

    It All Rises and Falls with Leadership

    Many would easily agree with the notion that long-term success for DevSecOps in an organization needs executive support and buy-in—and they would be correct—but real success needs much more than just budget approval and a shifting of resources. It needs ownership. It needs visibility. It needs truth-telling and that is where leadership comes in. According to the PuppetLabs “2018 State of DevOps Report,” 64 percent of C-Suite respondents firmly believed that security teams are involved in technology design and deployment, but only 39 percent of actual team members agreed. Bridging that kind of a gap requires breaking down traditional silos and evolving for better communication and transparency. The same survey also found that high-performing organizations were 24 percent more likely to use automated security policy configurations—a clear indication of tighter cooperation between security teams and the operations teams maintaining production environments.

    So, what are some of the key ingredients of successful leadership in this space? Glad you asked. Here are some things to look for:

    • Balance: Is the organization able to balance lowering technical debt with new work?
    • Shared ownership: Does everyone see how they contribute to business success and how their role impacts it? Is security included in that?
    • Shared learning: As projects progress and complete, is information moving across teams and projects? Are there best practices, lessons learned, etc., that are communicated both internally and externally?
      • Another way to think of this one: Is the level of “tribal knowledge” in your organization getting reduced?
    • Shared Vision Goals: Does everyone see and understand what matters most? Do they know the core metrics and KPIs, and how what they do impact them? Can they see their own success and bottlenecks?

    To have great success, you need leadership that defines the key standards, actively facilitates learning and collaboration and then provides the freedom necessary for innovation to occur. And there is another reason why you need it: Great leadership builds trust. Let’s explore that next.

    Your Speed of Delivery is Directly Proportional to Your Level of Trust

    The best depiction I have seen that correlates the speed of delivery with trust is this simple equation shared with me by DevOps Technical Evangelist Al Wagner from our mutual friend and Senior IT Specialist Lee Reid. It can also be found here. It simply states:

    Where “T” simply stands for time. In this simple equation, you can see that the HIGHER the percentage of trust is in the organization, the LOWER the overall value of the Time to Delivery will be, which is what we are all after. What that tells us is that today, businesses cannot tolerate a trade-off between speed and quality—you need both simultaneously.

    But how to you calculate trust?  In this model we are not talking so much about the work habits of the individuals as much as we are considering the outcome of each part of the process.  We  have to consider how often each of these areas “gets it right” versus how much we have to go back and rework.  Lee provides this example based on a client scenario:

    On a given project team the plan has about a 50/50 chance of being right, and the design is about 85 percent right, and the developers will get about 90 percent of the implementation right, and the testers have about 90 percent of the test cases right, and the release team has about 95 percent reliability of having the right stuff together to release, then your trust factor is something like

    .5 x .85 x .9 x .9 x .95 = 32 percent Trust

    Putting that back into the simple equation above, this level of trust means that your delivery time is three times longer than it could be.

    When my children were in elementary school and doing homework, I used to tell them that sometimes the best way to go fast is to go slow. The point was that if you wanted to solve problems you first had to understand them. Rushing to a formula or just trying combinations of numbers doesn’t work well and it ends up taking you twice as long to get done. This notion of trust and DevSecOps is similar and when you make the initial investment to build trust it can make a real impact.

    While it is true that adding application security to the Tplan and Tdeisgn parts of the process will initially increase those values, having application security as part of the software delivery process from the start—instead of at the end of a release cycle—will naturally cause the values for TDevelop, Tfix, and TEvaluate to shrink. This is largely because you are NOT having to go back to rework security into those areas midstream. It could also actually shrink Tplan and Tdesign over the life of the project for the same reason. In addition, the overall level of trust goes up because the risk that is associated with the delivered items decreases. We are more confident that artifacts are secure if vulnerabilities were identified early and fixed then.

    If you would like to learn more about how you can build trust, lower risk and make real business impact with application security, or to learn more about the potential impact of cognitive capabilities and artificial intelligence on your development efforts, download this complimentary Ponemon Institute study.

    — Rob Cuddy