devopsy.kinsta.cloud

Tag: devsecops

  • Continuous Security Through Developer Empowerment

    Continuous Security Through Developer Empowerment

    Every organization is embracing DevOps to one degree or another. The business impact of shipping software quickly and adapting to market needs is so immense that it has become a requirement—you’re either heading toward DevOps or heading toward bankruptcy. Yet, while our need for speed has increased, so has our need for security—and combining both is far from an easy task.

    The key to achieve the DevOps velocity is developer autonomy. Developers and their surrounding application teams need to make daily decisions about what to build and how to build it, and they need to do them on their own. Bottlenecks appear when teams are required to get external approval, slowing down the process and eliminating the dev team’s sense of ownership. The resulting effects are subpar outcomes in applications and attrition of top performing developers. 

    Yet, security practices, unfortunately, often depend on such reviews. Security audits and reviews are the bread and butter of most security organizations, requiring experts to determine if a decision is secure or not. For security to work at the speed of development, this practice has to change and give developers the power to make security decisions. The only alternative is insecure software and deceleration of software version development.

    How can we achieve such developer empowerment for continuous secure software development? Let’s review some key practices, and look at ops practices for inspiration. 

    Mindset Shift: From Super-Hero Operators to Multiplier Engineers

    Site reliability engineering (SRE) teams hold the same responsibility for a service’s uptime as the sysadmins that preceded them. The change in the role came from how they achieved this resilience. SREs don’t handle most incidents, nor do they review all code that affects reliability. Instead, they engineer the tools to empower developers to handle oncall and build operable software the right way. An SRE should only be pulled in for the extreme cases, be it an incident or architecture review. 

    Security teams must operate in the same manner. Their job must be tactical, working to reduce their involvement in the process and giving more responsibility to the developers. Cases will arise where their expertise is needed, but general inquiries should be kept to a minimum. 

    Tooling Change: From Security Tools to Developer Tools

    Before DevOps kicked in, app performance monitoring (APM) was owned by IT, who used synthetic measurements from many points around the world to assess and monitor how performant an application was. These solutions were powerful, but their developer experience was horrible. They were expensive, which limited tests developers could run. They excelled in explaining the state through aggregating tests, but offered little value to a developer trying to troubleshoot a performance problem. As a result, developers rarely used them. 

    Then, New Relic came on the scene, introducing a different approach to APM. Their tools were free or cheap to start with, making it accessible to all dev teams. They used instrumentation to offer rich results in developer terms (call stacks, lines of code), making them better for fixing problems. This new approach revolutionized the APM industry, embedded performance monitoring into dev practices and made the web faster.

    The same needs to happen for application security. If developers are the ones making impactful security decisions, then the most important users of security tools are no longer security folks—but rather developers themselves. Security tools should therefore focus on achieving developer adoption first and foremost, even when it comes at the expense of more effort by the security person involved. 

    Governance Change: From Blocking to Monitoring

    One of my favourite DevOps adages is, “If it moves, measure it. If it doesn’t move, measure it in case it moves”. It embodies just how critical it is to successfully monitor your running application. While CI/CD pipelines help you get software to production, monitoring gives you the confidence to do it at speed. You don’t need to worry so much about shipping a bug when you’re confident you’ll find out about it quickly and revert or roll out a solution.

    Security needs to embrace this approach. Finding out a security vulnerability has just been deployed and acting on it is almost as good as blocking it just before it was deployed—and far less disruptive. But to embrace this, you’ll need to invest in monitoring. 

    Developers are the first step in the application process. It makes sense that they are also the engineers of security solutions. Empowering developers to make security-first decisions allows IT teams and security professionals to address different parts of the business, saves time in the development process and ultimately creates stronger applications and outcomes. 

  • Enterprise Inertia Slowing Down DevSecOps Integration

    Enterprise Inertia Slowing Down DevSecOps Integration

    DevSecOps has become an increasingly popular topic as enterprises grapple with the continuing challenge of improving their cybersecurity infrastructure. As the first quarter of 2020 has come to a close, are we making significant progress in fully integrating the development, operations and security functions in the enterprise?

    The term “DevOps” was first used in 2009 by an IT consultant, Patrick Debois. Little more than a decade later, the IT industry is evolving to DevSecOps in response to the global cybersecurity threat and the realization that a more concerted effort is needed to effectively mitigate risk.

    One of the challenges is that DevOps itself is still not fully utilized in many enterprises, making those organizations more reluctant to embrace the idea of adding tighter integration with security teams and frameworks. Anyone who has worked in an organization knows implementing a cultural shift is difficult at best and not an overnight success. Moving from DevOps to DevSecOps does require such a shift.

    From Inertia to Integration 

    The Oracle Java 8 end-of-life story is a great illustration of the need to better integrate security into development and operations. Java applications traditionally were developed and delivered, and the inevitable security updates became the responsibility of the operations team to update the Java Runtime Environment (JRE) on each system that would utilize the application. This inevitably became a struggle of compatibility issues that resulted in exceptions versus resolving the security issues by updating Java.

    Java 10 resolves this struggle by creating a more symbiotic experience where the runtime components are baked right into the Java application as it is built. With this change, the dev team needs to take ownership of regular updates to the application to resolve security vulnerabilities, but this shift is slow in coming. Java 8 applications are still a critical part of many operations, even after the end of life of version 8, and companies now turn to OpenJDK and Corretto Java to continue in the legacy model.

    This is where inertia comes in. Enterprises stick to legacy systems, such as the Java 8 applications, and continue the cycle of compatibility issues preventing updates required to resolve security vulnerabilities. They have little or no attention from busy security staff, which long ago moved on to critical projects like protecting against the next WannaCry.

    Better integration–putting the Sec in DevSecOps more front and center–can be done.

    Here are some considerations to move from inertia to integration.

    Take Charge

    The concepts of ownership and responsibility need to change. A development team that has created a specific product needs to be responsible for taking care of future application needs, such as patching updates. The old-fashioned “We developed it; now we throw it over the wall to operations” doesn’t work in an integrated environment. The team needs to continue responsibility into the operation phase. This ends the volleying back and forth on who “owns” updates, and makes bugs, fixes and patching the DevSecOps team’s responsibility. 

    Get Your Track Shoes on

    Thanks to the plentiful options technology offers, enterprises are using varied development tools, browsers and vulnerability scans. This adds up to challenges since traditional patching is no longer always possible. At the same time, failure to execute updates in real time leaves the enterprise vulnerable to threats. For example, Microsoft introduced the new Edge Chromium edition which utilizes Google’s Chromium open source framework. Google had three Chrome releases between this year’s regular January and February Patch Tuesday events. This meant Microsoft had to integrate Chromium updates in real time or delay resolution to critical vulnerabilities for its customers. As another example, if an enterprise has an internal browser that used Chromium, the same updates have to be pushed out to protect users from threats.

    Think Totality of a Fix

    The best way to achieve integration is to ensure security is baked in as part of the development and operations process from the beginning. Before a new product/application is launched, a united DevSecOps team will be aware of all the components in that product, all the vendors who will have updates related to those components and the expected cadence of these updates. For enterprises updating a web or SaaS-based application, updates should be doable fairly quickly.

    Make Security First

    Time-to-market pressures abound but the strategic DevSecOps professionals are placing risk mitigation before operational impact. They are doing this in a landscape in which cyber hygiene continues to be an issue. Legacy systems are not being patched correctly; updates are being done too late, or not at all. To counter this, enterprises need to adopt a security first principle and improve the timeliness and thoroughness of patching and updates.

    Know Your Vendors

    A must-do is for DevSecOps to fully investigate what patching/updates a particular third-party vendor provides, what their release cadence is and what risk exposure it may present by not covering certain releases/companies. As part of this scrutiny, discern which vendors are responsive to security risks and which are not. With so many companies–Microsoft, Oracle, et al.–executing varied release schedules, adding automation into DevSecOps can further ensure updates are done in real time and cyber hygiene can be improved.

    Fulfilling the DevSecOps Promise

    DevSecOps can begin to reach its potential by paying attention to the people factor and using technology as a support rather than another headache. Whether an employee is in development, operations or security, they are working for the same organization.

    By introducing more automation into the process, gaining a clearer picture of all relevant components and the third-party vendor release cadence, and making security a participant at the beginning of new product development, smoother updates, better security and a productive culture will result.

  • ClickShare Vulnerabilities May Have Been Patched, But They Mask a Much Bigger Problem

    ClickShare Vulnerabilities May Have Been Patched, But They Mask a Much Bigger Problem

    I think we can all recall a time in recent memory where, in a meeting or at a conference, someone has had issues with presentation technology. It happens so often that there is almost an expectation of a clunky experience, at least initially. It stands as no surprise, then, that ClickShare’s seamless app was immediately popular with end-users. For them, there is nothing easier than using a ClickShare application to push a presentation from their laptop, tablet or smartphone over to a big screen or conference room projector.

    Belgian-based digital projection and imaging technology supplier Barco designed their automation platform to work that way, and big business embraced the concept. FutureSource Consulting puts Barco’s market share in conference technology at 29%, with integration into 40% of all Fortune 1,000 companies.

    When researchers from F-Secure revealed in December that the seemingly innocuous automation platform was riddled with security vulnerabilities, it sent shockwaves through the business community. The uncovered security flaws are critical in nature, and could potentially enable any number of malicious activities. 

    Researchers demonstrated how the vulnerabilities could allow remote users to snoop active presentations, create backdoors into secure networks or even configure a spyware distribution server that would infect every user connecting to a Barco device. Suddenly, companies were faced with the prospect of having serious security problems installed directly inside conference rooms and offices throughout their organization. And because of the nature of the vulnerabilities, a single compromised device could support a network-wide breach.

    “An attacker who successfully compromised one unit gains the ability to decrypt as well as produce valid encrypted images for any unit, whether within one family or across families,” wrote F-Secure officials in their report. “Further, such an attacker may gain access to sensitive data at rest such as the configured WiFi PSK and certificates.”

    To its credit, Barco has been extremely proactive in issuing patches and fixes to the vulnerabilities found within its products. Security vendor Tenable recently released a report showing 15 vulnerabilities across eight presentation tools, including Barco. As of February, only Barco has been active in deploying fixes.

    Although some of the Barco vulnerabilities require hardware changes (and these will be a nightmare to deploy, if a firm acts to this degree to secure them at all), many of them can be corrected with software patches. This gives most enterprise users a seemingly good plan to fix their immediate problems, but they are hardly in the clear now. The problems with Barco are just the tip of the iceberg when it comes to dealing with vulnerabilities in well-known hardware and software products.

    The Root of the Problem

    Now that the immediate issues have been resolved, we need to ask how devices with serious security flaws ended up sitting in thousands of conference rooms worldwide, or why they were so poorly designed and programmed in the first place. It’s not like the F-Secure team was uncovering zero-day or previously unknown vulnerabilities. Ten of the flaws discovered in the Barco products were associated with well-known, common vulnerabilities such as code injection attacks. Most already had Common Vulnerabilities and Exposures (CVE) identifications.

    So how did decades-old CVEs get coded, or even hardwired, into modern presentation tools? The only possible answer is that developers either didn’t know about them, or that security was not made a priority as the Barco devices were being designed. Sadly, this is a common situation, and certainly not exclusive to the Barco teams.

    The best time to fix a vulnerability is while an application is being developed, long before it gets sent out to users. The worst (and most expensive) time is after a product has been deployed, or after it’s been exploited by hackers. This can be a hard lesson, and one that Barco will surely learn as its once-impenetrable market share takes a hit following this security fiasco.

    Shifting security fixes back toward the development process isn’t easy, but is necessary in today’s world where even seemingly simple devices like presentation tools are both surprisingly complex, and also networked into everything else. In this environment, security must become an organizational best practice. It doesn’t matter if a company is programming apps for social media or manufacturing smart toasters, security must be considered in every facet of an organization.

    Prioritizing security best practice and making it a shared responsibility is the goal of the DevSecOps movement, where development, security and operations teams work together to code and deploy secure software and products. It requires as much of a change in culture as anything else. The new mindset needs to be that deploying a working product with security vulnerabilities is just as much a failure as creating one that can’t perform its primary function.

    In a healthy DevSecOps environment, anyone touching software should be security-aware, with developers receiving relevant and frequent training to avoid introducing disastrous bugs into their work. If the teams working for Barco had looked at security as a shared responsibility, there is no way that such a large collection of vulnerabilities, including decades-old CVEs, would have made it into their presentation tools.

    The Safe Path Forward

    Nobody wants to be the next Barco, having to explain why well-known security flaws were deployed through their devices to thousands of enterprise networks around the world. To avoid that fate, companies developing software or smart hardware should immediately prioritize security as both a shared responsibility and an organizational best practice. Creating a healthy DevSecOps program will take time and likely also require a shift in culture, but the results will be more than worth the effort. Robust DevSecOps can crush vulnerabilities long before they cause trouble.

    For companies buying products and software, it’s in their best interest to support firms that have embraced DevSecOps. Doing so will go a long way to making sure that the devices and software obtained from them aren’t ticking time bombs waiting to be exploited by increasingly skilled attackers.

  • Digital.ai: The Companies Formerly Known As XebiaLabs and CollabNet Plus

    Digital.ai: The Companies Formerly Known As XebiaLabs and CollabNet Plus

    Another shoe has dropped in the continuing story of TPG Capital‘s foray into DevOps. The company formed by the merger of XebiaLabs and CollabNet VersionOne is now Digital.ai. Additionally, the company has acquired Arxan, a leader in application protection.

    Digital.ai logo

    I had a chance to catch up with Digital.ai’s CEO Ashok Reddy and Arxan COO Mark Lorion, who will be Digital.ai’s COO of Platform going forward. The video of our discussion which is also available on TechStrong.TV, appears below.

    What They Said

    Reddy made it very clear that Digital.ai is not content with simply adding this AppSec company to the mix. It is actively involved in due diligence operations on several other acquisitions. Reddy and the TPG team feel that the current COVID-19 situation affords those with cash to pick up some bargains. That may have been the case with Arxan, which was owned by private equity firm TA Associates, which acquired Arxan in 2013 for $132 million. TA may have decided it was time to get off the merry-go-round, given the unsettled state of things right now.

    In any event, TA’s loss is Digital.ai’s gain. Arxan adds some much-needed security muscle and a blue-chip bank customer list. But, of course, the real play here is building out the end-to-end DevOps company that will compete with GitLab, GitHub, CloudBees, JFrog and others. We detailed this market in an article when the original merger was announced.

    My take is that indeed Digital.ai is not a complete company just yet. It may not even be done in the DevSecOps space, as Arxan is not your typical DevSecOps technology. It may have been more of an opportunistic acquisition.

    Also, in terms of executive team, Reddy has been quietly assembling a team made up of some of the best people he worked with at previous stints at IBM, CA/Broadcom and the executives from the acquired companies.

    Stay tuned for more on Digital.ai’s continued evolution in the months to come.

    — Alan Shimel

  • 2020 DevSecOps Community Survey: Of 5,045 Developers, Guess How Many Are Happy?

    2020 DevSecOps Community Survey: Of 5,045 Developers, Guess How Many Are Happy?

    We just released the seventh annual DevSecOps Community Survey. We captured the pulse of over 5,000 developers in 102 countries. What we learned about trends in DevSecOps will surprise you—especially the parts about how happiness influences company culture, software security and business competitiveness.

    When we tallied up the results we quickly saw some interesting correlations. Organizations with mature DevSecOps practices innovate faster and produce more secure software. Why? It boils down to having happier developers. Developers in mature practices were 1.5x more likely to say they enjoyed their work. Mature DevSecOps practices are also 1.8x more likely to use automated security tools, which certainly makes developer’s lives easier.

    Here are just a few interesting tidbits we discovered.

    Happiness Influences Culture

    “We’ve always known that DevSecOps is about culture. The 2020 DevSecOps Community Survey, for the first time, reveals clear and convincing empirical evidence that developers are happier and more productive when security is part of the digital transformation and DevOps journey,” said James Wickett, head of research at Verica.io.

    Just like you can’t buy happiness, you can’t buy good culture. You have to cultivate it. Here’s what the survey showed:

    • Developers who received secure coding and programming opportunities are five times more likely to enjoy their work.
    • By contrast, only 19% of unhappy developers get similar training. Unhappy developers say they aren’t satisfied at work and are actively seeking other opportunities. You can see why they are grumpy cats.
    • Happy developers don’t feed off rumors—at least, not for security incidents. Instead, they focus on empirical evidence from better integrated tooling and security teams. Developers working in mature practices are 3.8x less likely to rely on a rumor.

    Happiness Influences Security

    “Mature DevOps practices are constantly testing, deploying and validating that software meets every requirement and allows for fast recovery in the event of a problem. As a result we can easily say, ‘DevSecOps is DevOps done right,’” said Hasan Yasar, technical director and adjunct faculty member at the Software Engineering Institute, Carnegie Mellon University.

    Happy developers build more secure code. Here’s how they do it:

    • Mature DevOps practices are 1.7x more likely to have a complete SBOM (Software Bill of Materials), including dependencies. This makes finding and fixing open source components faster and easier. 
    • Code quality? Check. Happy developers are 3.6x less likely to neglect security. This is because, with proper tools, they can focus their efforts on what is important.
    • Grumpy developers are 2.6x more likely to ignore a security warning compared to happier developers working in more mature DevSecOps practices.

    Happiness Influences Business Outcomes

    “We cannot achieve higher levels of DevOps maturity until we understand how tightly woven people are into the transformation process. More than anything, DevSecOps success is tied to human effort,” said Jayne Groll, CEO of DevOps Institute.

    Businesses depend on secure code built by happy people. Look at these findings:

    • Happy developers are 1.4x more likely to follow their company’s open source governance policy. This is an important part of risk management. Ignoring policy and lessening software supply chain security can have catastrophic business implications.
    • Happy developers are 1.8x more likely to recommend their employer to peers and friends, a real boon for attracting and retaining talent in a competitive environment.
    • Happy developers are 1.7x more likely to get the job done. If productivity is the name of the game (and it often is) then businesses with mature DevSecOps teams outcompete the rest.

    Download the full report here. Also, be sure to register for All Day DevOps | Spring Break on April 17th. I’ll be doing a session that goes over the report in depth and you won’t want to miss it.

    — DJ Schleen

  • DevOps Chats: DevSecOps and OpenShift, with Red Hat

    DevOps Chats: DevSecOps and OpenShift, with Red Hat

    In this DevOps Chats, we had a talk with Kirsten Newcomer, senior principal product manager at Red Hat. It is a great discussion on the state of DevSecOps and how the Red Hat OpenShift team is trying to make it easier for developers, DevOps and cyber folks to work together to create more secure applications.

    As usual, the streaming audio is immediately below, followed by the transcript of our conversation.

    Transcript

    Alan Shimel: Hey, everyone, it’s Alan Shimel, devopsy.kinsta.cloud, Container Journal, Security Boulevard—you’re listening to another DevOps Chat. Welcome, and I’d like to welcome to our DevOps Chat Kirsten Newcomer of Red Hat. Kirsten, welcome.

    Kirsten Newcomer: Thank you, Alan. Great to be here. So, I’m Kirsten Newcomer as you said, Red Hat. My focus at Red Hat is DevSecOps for cloud-native applications, been working at Red Hat for about four years now. It’s really kind of fun. I’ve managed to leverage kind of my—all sorts of different elements of my background over the years. I spent time at Rational Software, nine years, working on developer tools.

    Shimel: Sure.

    Newcomer: Yeah. Spent some time with BMC BladeLogic on the operations side of the world, seven years at Black Duck Software focusing on solutions that help companies manage vulnerabilities in open source software, and that led to the move to Red Hat where I get to focus on OpenShift security and DevSecOps and sort of everything comes together. So, DevOps—

    Shimel: It really does.

    Newcomer: Yeah.

    Shimel: I mean, following your career, I think I know just about all of those companies intimately and know a lot of people—I’m sure we even have a lot of people in common that we’ve worked with.

    Newcomer: I bet we do.

    Shimel: It’s funny. So, you were at Rational, left Rational only to return to IBM Rational.

    Newcomer: I know! [Laughter] That is true.

    Shimel: You know, it’s the circle of life right there, man. [Laughter]

    Newcomer: It is the circle of life, yep.

    Shimel: Welcome. So, Kirsten, look, DevSecOps is of course something near and dear to us. We put on every year at the RSA conference our DevSecOps Days or DevOps Connect/DevSecOps Days.

    Newcomer: Uh-huh.

    Shimel: We actually have the virtual event of that live on Thursday, I think it’s the 26th, March 26th.

    Newcomer: Nice.

    Shimel: So, you know, it’s something, quite frankly, this whole security thing is the reason I got into DevOps is, I thought—what a great opportunity for security to cure a lot of original sin.

    Newcomer: Absolutely. [Laughter]

    Shimel: But anyway, you know, when we look at the DevSecOps and the DevOps world, Kirsten, you can’t help—you must take notice of the impact that Kubernetes and the whole cloud-native kind of movement has had on this.

    Newcomer: Okay.

    Shimel: And a great—

    Newcomer: Absolutely, no—big changes, yeah.

    Shimel: And, you know—and it’s really at the core of so much of what is being done right now from an infrastructure point of view. And oftentimes, I don’t know if people—like everything else, right? I remember when the cloud came out and we moved from client-server and all this. Security, there’s always that lag, right, where, “Oh, yeah, we gotta do security with this.”

    Newcomer: Yep.

    Shimel: And we had a little bit of that with Kubernetes, I think, where—

    Newcomer: No, I think that’s fair.

    Shimel: Yeah? You know, people were rushing to do Kubernetes and experiment with it, deploy it, and then someone started saying, “Wait a second, what about security, here? What about compliance? What about this or that?”

    Newcomer: Yeah.

    Shimel: And now, I will tell you, and I’m interested in your opinion—I thought the lag time was much shorter this time than what we’ve seen in the past.

    Newcomer: Oh, I absolutely think so. And I think it’s really interesting. A couple of things that I’ve observed. One is that I really think containers and Kubernetes have created a terrific opportunity to shift security left, and I’ll kinda be more explicit about that in a couple ways. But I, in fact, have worked with a chief of cyber defense in the public sector who believes and evangelizes that containers improve security, and I’ve stolen his line, because I just actually think that that’s really valid.

    Because of the changes in the model, with the OS dependencies and the run time, everything’s kinda packaged together and you have that immutable container image that you’re always deploying from, you really can manage things in a new way and not every team is ready for this, and not every team, not every security vendor is ready for this, either.

    So, I think the opportunity has been twofold. One is to kind of adjust the thinking about security so that it no longer becomes something that happens at the end, but that’s still a process, that’s still a lot of work. And the other thing is, the security tooling space itself has historically been pretty siloed, right? There’s been, you know, the OS layer security, the network layer, there’s been kinda—you know, my SIEM and my SOC that I’m working with, and my cert management and my vaults.

    And none of them really were designed to work with the level of automation that Kubernetes and containers enable, which led to this really interesting opening for smaller companies, a lot of startups that now have been around for a little while, actually, to start breaking down those silos and delivering security capabilities in a new way that kind of really do fit the DevSecOps model, because you get kinda one vendor bringing a set of capabilities that ranges from CI/CD integration to run time security to network security, really fascinating space to see how it’s been evolving.

    Shimel: It really has. The other thing that I will tell you, Kirsten, that I was again encouraged by was when I first started seeing people talk about security through containers and Kubernetes, it really was like—let me scan your container for vulnerabilities, right?

    Newcomer:  Yeah. [Laughter]

    Shimel: And I’m thinking to myself—I don’t care whether my code’s in a container or in a hypervisor or on bare metal. A vulnerability scan is a vulnerability scan. There’s gotta be more to this than just a more vulnerability scan kind of a thing.

    Newcomer: Absolutely. And that’s, I think, where, as you mentioned earlier, that, you know, Kubernetes really has come a long way in that space. So, there are things like, if we think about—so, absolutely. Vulnerability scans are fundamental and kinda, people always understood those. They tended—historically, they’ve happened a little too late in the life cycle, making challenges for the AppDev team and such.

    We’re seeing that shift left a lot, but then there’s all these other range of things that you should be doing and can be doing to build security into the platform. And so, when we saw pod security policies in Kubernetes, for example, that’s a way that the Kube admin can take advantage of the Linux features that enable container isolation at the Kubernetes layer and enforce things like ensure that a container doesn’t run with unnecessary privileges.

    Shimel: Sure.

    Newcomer:  And so, we are seeing more capabilities—now, that’s still beta, so the community still has some work to do there. But as you say, we’re seeing more emphasis and, you know, the open source Kubernetes security audit sponsored by the CNCF is a great example of that.

    Shimel: Oh, absolutely. And look, CNCF has done yeoman’s work here. They’ve really, really done a nice job with the whole Kubernetes project, specifically security.

    You know, but one of the reasons, like—we did a webinar yesterday for instance and it was on, you know, the different cloud providers versions of Kubernetes and—

    Newcomer: Uh-huh.

    Shimel: [Cross talk] environments. And, you know—fantastic, we had 700 and something people sign up for this webinar.

    Newcomer: Great.

    Shimel: A lot of great questions, even on the YouTube afterwards, a lot of great questions. One of the things that became clear to me in listening to people talk is that—you know what? Just going to, whether it’s AWS or Azure or GCP or whatever and setting up my Kube environment may not be the best thing for me because I’d rather get one of the package distributions, if you will, or a more complete solution, of which Kubernetes is a piece, like an OpenShift, but some others were mentioned—Rancher and so forth.

    Newcomer: Sure.

    Shimel: But OpenShift was the most popular one mentioned. And I think one of the reasons is, people want to know that security has been kinda built into it.

    Newcomer: Yeah.

    Shimel: Beyond just the Kube release, whether it’s 1.4, 1.5, what have you, 1.6—you know, what else has been wrapped around that just K8—

    Newcomer: Yeah.

    Shimel: —to make it easier?

    Newcomer: Absolutely, and that’s been a big focus for us at Red Hat, really, from the beginning. You know, an enterprise solution—and this is a place where, so, we’ve talked about changes in the security tooling that needed to be made to adjust to the technology, but by and large, the principles still apply. You know, you still need audit logs, right? You still need, you know, to log data from the cluster itself. You need log data from the applications. You need monitoring tools. And so—and you need to think about host arrest security as well.

    And so, we’ve kind of really looked at the stack as kind of spanning all of those things, and we want to make it as easy as possible to manage that solution as a solution, right? And so, that if you—and plenty of folks do this, you know, and get value from it. But if you’re in a situation where you can afford to kind of do a DIY Kubernetes and add in a logging stack and add in kind of all the different pieces that you really need for the enterprise solution, then you have to maintain them each separately and you’re maintaining the host OS separately as well.

    And so, we’ve really focused recently, especially with OpenShift 4, on automating everything, which includes the host operating system. So, Rel CoreOS, Fedora CoreOS available as an open source project. It’s a container optimized operating system that we manage as part of the full platform. It makes it much easier to apply OS updates across your cluster. You can do that in a way that has zero downtime for your well-behaving apps. You can take a node out of service in the way you would, you know, that Kubernetes manages it. Update that with a patch to OpenShift or its components or the operating system itself, put it back in service.

    And so, kind of a combination of looking at the whole stack holistically, what is an enterprise need in order to ensure that they have a secure environment, a stable environment, a highly available environment and making sure that we kind of leverage the terrific declarative and automation capabilities of Kubernetes kind of throughout the whole stack.

    Shimel: Absolutely. You know, I didn’t realize just really all of—you know, until you just said it, I never kinda put together all the pieces of what you guys are doing with that. It’s pretty amazing, isn’t it?

    Newcomer: It’s really cool. And one of the other fun things that we get to do is, we actually—so, back to, if we think about the host OS layer for a bit, right? If you need to scale your clusters, that means adding a new host and you’ve got the work involved to be sure that you deploy that infrastructure and that you secure that infrastructure according to your standards and then you put Kubernetes on top of it and add it to the cluster and you’re ready to put workloads on it.

    And we’re using the concept of Kubernetes operators to automate the components of OpenShift, including the host OS. So, we have a machine config operator, so if you’re deploying OpenShift in a cloud where you can automate the addition of infrastructure easily, you can use the machine config operator and the declarative nature of Kubernetes, right? It’s a container optimized OS, it’s delivered primarily as container images, the user space is read-only, you’ve got that declaration of what a host should look like and you can just use the machine config operator to spin up new hosts and add them to your cluster.

    Shimel: Excellent. You know, we would be derelict in our duties, I guess, if we didn’t mention the whole COVID-19 and how that’s affecting things.

    Newcomer: Fair enough, yes.

    Shimel: In my mind—so, first of all, I don’t think a lot of what we’re seeing right now and its effects are necessarily three-week or six-week or, you know, a lot of this stuff is gonna stick.

    Newcomer:  Yeah.

    Shimel: I think one of the things that are gonna stick are, people are gonna want more automation, right?

    Newcomer: Absolutely.

    Shimel: In their deployments, in their operations, in their infrastructure.

    Newcomer: Yep.

    Shimel: And automation, from a security point of view, is kind of a double-edged sword, right? At least we, from—when things are automated from a security point of view, we think we know exactly what’s gonna happen, and so we can plan around that. On the other hand, security people tend to get a little fritzy, you know, when things come out of our cold, dead hands, here.

    Newcomer: Yeah.

    Shimel: What do you think? How—what do you think?

    Newcomer: I know; I agree with you. And I think one of the things that, one of the ways that Kubernetes is kinda pushing the boundaries with security teams. And I talk to a lot of different security teams over time. So, it’s kinda like—you know, they’re a little bit more comfortable with the idea of DevOps for the application layer at this point, right? For those containerized apps that are gonna be deployed, it’s like—okay, they kinda have an idea of, “Yep, security static analysis tools, integrated vulnerability scanners integrated.” There’s some cool new stuff coming out that I’m seeing on the market, they’re tools that are starting to evaluate the configs and the Dockerfiles, too, that’s really great.

    But—and then as we continue to move into the ops space, right, with Kubernetes and the SDN, which is required for any Kubernetes cluster, and now we have service mesh added into the mix, or Istio for microservice based communications. The network security teams are uncomfortable with who’s gonna manage network policies at the Kubernetes layer and now, I’ve got service mesh or Istio policies as well, and who’s responsible for those?

    And in the end, it’s really about visibility. So, if we circle back to automation, we’re really looking at infrastructure as code, you know, GitHub’s model that applies again to the Kubernetes layer, to the configuration of the Kube cluster as well, and we need to put in place, you need the automation to get the best results, but you need to do it in a way that gives the security team visibility into the kinds of things they’re used to seeing.

    And so, I think this is another way where we’re starting to shift left in new ways, and I’m also seeing solutions that help automatically generate network policies based on evaluating an environment or an automatic generating service mesh policies based on evaluating the config for the apps.

    But all of that, you know, the network security guys aren’t necessarily gonna log into the cluster to look at that. So, how do they get the view they need to get, and how do they help the AppDev teams understand that whole risk management and their security perspective? And so, this is where it’s not just about tools, it’s about people and processes, right? And I think there’s so much richness here, but you’re absolutely right—we can’t do it without automation. And so, we’re gonna have to help these people become comfortable with this shift while ensuring that they get the visibility they need to know that things are being done in a way that meets their risk guidelines, their regulatory requirements, and the security posture they want.

    Shimel: So, to me, one of the fundamental things with DevSecOps, one of the fundamental shifts in DevSecOps is, first of all, security people recognizing that they’re not the only ones who care about security.

    Newcomer: Mm-hmm.

    Shimel: Believe it or not, the developers and the DevOps folks, they do care about security. They may not be security trained people, but they care about security.

    I think the second thing with that is, the developer people have to realize, and the DevOps teams have to realize that the security people aren’t there just to be the roadblock.

    Newcomer: Right.

    Shimel: And they aren’t always gonna be the people who say no.

    Newcomer: Right.

    Shimel: And what we’re—so, this has nothing to do with technology, per se. This is a people to people issue.

    Newcomer: Absolutely.

    Shimel: And I think we’re making real progress in that, and as a result, what we’re seeing is security people paying for and approving security tools that developers in DevOps teams are using—

    Newcomer: Yeah.

    Shimel: —on the Kubernetes platform—

    Newcomer: Yep.

    Shimel: —you know, because that’s kinda the platform of choice now, and that, to me, that’s what keeps me optimistic about this.

    Newcomer: Yeah. No, I’d agree, and also, when we think about the broader landscape, I mean, there’s, the security threats are just ever-evolving, ever-changing. And there aren’t enough security trained professionals to go round, right? There aren’t enough cybersecurity folks, et cetera.

    And so, we have to figure out, how do we help teams scale, and collaboration is the way to do that. And you’re right, I see the same thing you do, right, that the security team is approving those tools that are container native, Kube native tools that help to ensure. And in some ways, I don’t mean this in a negative way, but in some ways they kind of are pushed into that because they’re traditional tools that used to be used kinda at the back end of things aren’t effective in the container and Kubernetes world.

    And so, they really do have to go with this shift that we’re seeing in the vendor landscape, and it just opens up this terrific opportunity, though, for that collaboration and that shared knowledge which, you know, is an adjustment for people sometimes.

    Shimel: Yeah, agreed, agreed. Kirsten, when we first started, I said, you know, the time goes really quick, I’m only gonna keep you here 15 minutes—well, that was 21 minutes ago.

    Newcomer: [Laughter] Okay.

    Shimel: We—yeah, we did it. Alright. We’re gonna have to call a wrap on this one. Maybe we’ll have you come back another time and we—

    Newcomer: It’s a plan!

    Shimel: —can continue the conversation there, good?

    Newcomer: Great. It was great fun—thank you so much, Alan.

    Shimel: Thank you.

    Newcomer: Alright.

    Shimel: Kirsten Newcomer, Red Hat, our guest here. This is Alan Shimel for devopsy.kinsta.cloud, Container Journal and Security Boulevard. You’ve just listened to another DevOps Chat. Stay well, everyone.

    — Alan Shimel

  • HashiCorp Strengthens Vault Secrets Management

    HashiCorp Strengthens Vault Secrets Management

    HashiCorp has released an update to its HashiCorp Vault platform for managing secrets and encrypting data that now includes an internal storage option that eliminates the need to manage a separate storage backend.

    Version 1.4 of Vault also adds an OpenLDAP Secrets Engine, which automates the management of static entities within OpenLDAP directories, and the ability to authenticate users and applications using the Kerberos authentication protocol.

    HashiCorp has also updated the enterprise edition of Vault to add support for a Transform Secrets Engine that can be used to secure data that reside outside of Vault, such as credit card numbers, by applying tokenization techniques to mask data, as well as a Vault Helm Chart tool that makes it easier to deploy Vault on Kubernetes clusters to the enterprise edition of the tool.

    Finally, the enterprise edition of Vault adds improved disaster recovery capabilities and support for NetApp enterprise key management to more easily manage encrypted data.

    Chris Kent, director of product marketing for HashiCorp, said thanks to the increased focus on data privacy there is a greater appreciation for the need to maintain application secrets in a platform that resides outside of the core application. Regulations such as the General Data Protection Rule (GDPR) enacted by The European Union and the California Consumer Privacy Act (CCPA) penalize companies for losing control over credentials and data. As a result, more organizations are looking for a way to better manage application secrets, noted Kent.

    At the same time, cybercriminals are becoming more adept at discovering the credentials of end users that developers sometimes inadvertently leave accessible within their applications, he noted. Vault provides a way to manage and maintain application secrets outside of the application in a way that serves to significantly enhance overall application security.

    As developers and cybersecurity teams collaborate more closely as part of any effort to embrace best DevSecOps processes, it’s becoming apparent that a lot more attention must be paid to how application secrets are managed. In fact, arguably one of the biggest impediments to improving the overall security of applications is the way developers have historically managed application secrets. It’s not uncommon for cybercriminals to compromise an application in a way that gives them access to thousands of individual user names and their associated passwords, many of which unfortunately are reused both inside and outside of the extended enterprise. The challenge at this point is not so much making the tools available to manage application secrets as much as it is changing the processes and culture within organizations that are just now starting to embrace DevSecOps.

    In general, as organizations become more proficient at managing application secrets the more secure application environments should become. Of course, secrets management is only one aspect of application security. However, the absence of an effective strategy for managing application secrets can prove catastrophic. By the time most IT organizations realize their application secrets have been compromised, it’s usually far too late to undo the damage.

    — Mike Vizard

  • The Increased Role of Colocation Providers in Data Centers

    The Increased Role of Colocation Providers in Data Centers

    You would be hard-pressed to find something more complex than a modern data center. A state-of-the-art building stocked with all the necessary software for a major company might run into billions in costs if you decide to build and operate your own from the ground up.

    Even though many pundits had predicted data centers would become a thing of the past due to the ever increasing adoption of SaaS, as we previously predicted, data centers are still popular with companies and evolving for 2020. One phase of that evolution is for businesses to box up all of the gear in their data center and install it with what is called a colocation provider, or “Colo” if you’re looking for a snappy acronym.

    “Going Colo” refers to renting space in a building, usually a rather large building, and moving all your physical servers and network hardware there. You’ll have neighbors who are doing the same thing because this is just a fancified way to say you’ll have a landlord.

    The Colo market is getting big in a hurry. The global market was $31.5 billion for this type of data center arrangement in 2017, but is expected to expand to at least $62.3 billion by 2022. What a company, large or small, has to ask itself is if this type of arrangement is a good thing for DevOps. Let’s ponder that.

    It’s Cheaper When You Don’t Own the Building

    If you remove the possibility that you could be thrown out of the space at a moment’s notice (a worst case scenario that a good rental agreement should address), the idea is a no-brainer from a cost savings perspective.

    For an enterprise, let someone else take on the risk of construction and infrastructure maintenance. For a small-to-medium-sized business, the cost of building and monitoring a modern data center makes it impossible.

    The unfortunate reality is that, when you do own the building, someone has to take care of it. In a perfect world, every company would have a dedicated IT staff; if you’re Microsoft, Apple, Google or Facebook, that certainly is the case.

    For much of the tech world, the choice to maintain a data center onsite often results in personnel being frequently dragged out of their role in DevOps and asked them to do something related to the data center, often a task they are not qualified for. This creates holes and inefficiency in the development process. That’s a bad thing. 

    Good Infrastructure Means Better Uptime

    Guess what’s also bad for DevOps? The team’s least favorite scenario—besides an unexpected meteor strike—would be system crashes and downtime related to hardware issues. Downtime can be incredibly expensive for a multitude of reasons besides the obvious—employees sitting around twiddling their thumbs. There is also the scenario of actual lost income. As Amazon found out the hard way, the number can be astronomical.

    With no internet connection, business payment systems, no matter how sophisticated, are likely to be non-functional, though some will queue payments and run them through when the system goes back online. This is why it’s a good idea for businesses to find ways to speed up transactions from clients and customers, such as automating invoices and customizable invoicing tools that allow multiple options for straightforward payments.

    This is also another area where going Colo can be a huge help to development. Think about it. In return for scratching a monthly rent check, you get diversified and redundant power sources, the most reliable internet on the planet and enough bandwidth to strangle an elephant.

    The bottom line is that moving equipment to a Colo space should be accompanied by a significant reduction in downtime. If it’s not, you need a different landlord. Downtime is the enemy of product/service development progress. A modern Colo arrangement means you don’t have to spend time running down to the local Dollar General in search of an extension cord when one flames out. Okay, you really shouldn’t be using extension cords from a dollar store in your data center, but you get the idea. 

    Secured Data and Premises

    The only type of downtime worse than that caused by insufficient infrastructure is when data is breached either through hacker online trickery or physical penetration of the premises. The latter happens more than you might think. Bad guys decide to go old school and break into a Colo center in the middle of the night to physically drag servers out of the rack and spirit them away.

    With the intense global focus on thwarting hackers, DevOps is already in the process of evolving into DevSecOps as we speak. The reason is simple. Development teams are realizing that a focus on security from the start reduces the chance that a project gets interrupted mid-development resulting in—you guessed it—downtime.

    With a Colo provider, let them worry about security.

    In a Word, Scalability

    With the everyday lexicon expanding to include cloud storage possibilities, the idea of scalability is at least somewhat familiar to the masses and certainly to businesses.

    Google actually started out in a colocation data center before their data collection grew to approximately the size of the Milky Way galaxy. For IT in general and DevOps in particular, scalability is the Holy Grail because no company expects to stay the same size forever.

    A smart DevOps team, especially one that works in any of the growing number of SaaS business models, will put the bug in management’s ear about the benefits of Colo long before the need arises.

    The worst time for an upwardly mobile small or medium-sized company to start thinking about making the switch is after you’ve already outgrown present resources. For example, when working with inadequate bandwidth, a peak time data spike can slow development as local computers struggle to find enough bandwidth to function.

    Another dirty little secret is that Colo providers typically get better internet rates than a single company, so there’s that. With the cost of a spike spread across all tenants, the actual cost to you will be considerably less than trying to manage your own data.

    Final Thoughts

    The bottom line is that the service provided by a Colo center should reduce the strain on a company’s IT budget and resources. Common services provided to a tenant are support personnel, who monitor and troubleshoot systems and issues before they grow into a big, fat calamity. Money you don’t spend providing basic service to an onsite data center is money you can funnel toward doing your company’s main work.

    There’s one more thing a Colo site provides that shouldn’t be undervalued, and that is peace of mind, because you have removed much of the stress, cost and personnel drain that accompanies running your own data center. Farm it out. Let someone run it who hires staff with the particular skills needed and has a vested interest in keeping you happy.

    — Gary Stevens

  • Palo Alto Networks Advances DevSecOps in the Cloud

    Palo Alto Networks Advances DevSecOps in the Cloud

    Palo Alto Networks has updated its Prisma Cloud to make it easier to embrace best DevSecOps processes while deploying workloads in the cloud.

    New capabilities include the ability to define and apply cybersecurity policies to continuous integration (CI) and continuous delivery (CD) workflows and scanning tools for discovering misconfigurations in cloud infrastructure templates, which is the most common source of cloud security issues.

    Palo Alto Networks has also added the ability to scan virtual machines native to the Amazon Web Services (AWS) before they are deployed as well as an ability to apply policies to workloads running on the AWS Lambda serverless computing framework via a single click. That capability eliminates the need to manually install wrappers in application code running on the AWS Lambda framework.

    John Morello, vice president of Product Management, Container and Serverless Security at Palo Alto Networks, said in the wake of the COVID-19 pandemic, the rate at which workloads will move to the cloud will accelerate. Deploying applications allows organizations to retain a higher level of flexibility, which, given the current uncertainty over how long the COVID-19 pandemic may last or even return once this latest outbreak is contained, means organizations need to be able to manage workloads centrally from any location where their IT staffs are located.

    Of course, cybercriminals that have become especially adept at scanning for cloud misconfigurations realize this as well. As such, many of them will be focusing their future efforts on scanning for vulnerable workloads residing on public clouds, Morello noted.

    A recent analysis of public cloud configurations published by the Unit 42 research arm of Palo Alto Networks found more than 199,000 templates that had medium-to-high vulnerabilities. The most vulnerabilities were discovered in templates created using CloudFormation (42%), Terraform (22%) and YAML for Kubernetes (9%).

    As cloud computing environments become more complex the opportunity only increases for misconfigurations involving, for example, ports left open. IT organizations are increasingly trying to secure a wide range of application workloads running on a mix of public cloud services based on virtual machines, Kubernetes clusters and serverless computing frameworks. Public cloud computing environments assume that responsibility for cybersecurity is shared between the cloud service provider and the IT teams who employ those services. Unfortunately, developers often assume the cloud service provider is assuming more responsibility than they actually do. Most cloud service providers are really only promising to secure the infrastructure they manage. Responsibility for application security remains firmly in the hands of the IT teams deploying cloud applications.

    To rise to that challenge, many organizations have been embracing best DevSecOps processes that shift cybersecurity responsibility further left toward developers. However, to achieve that goal, developers need access to tools that are well-integrated with the CI/CD platforms that DevOps teams rely on to push code out to cloud platforms. Obviously, that shift represents a substantial cultural change that will take time to fully manifest within most organizations. However, no such change is ever likely to be attained without being armed with the tools needed to enhance application security within the context of their existing CI/CD workflows.

    — Mike Vizard

  • Filling the Skills Gap for Effective DevSecOps

    Filling the Skills Gap for Effective DevSecOps

    With the rise of DevSecOps comes a whole new need for training and upskilling. It isn’t a secret that the cybersecurity skills gap will only continue to grow. The security skills gap combined with the rise of DevSecOps has introduced hidden challenges organizations must address.  

    The Rise of DevSecOps 

    While DevOps accelerates agility and scalability of organizations, it may also expand risks of threats—particularly if security is an afterthought in an organization’s cloud strategy. In a recent survey, 52% of companies admitted to scaling back security measures to meet a business deadline or objective. Similarly, 68% said their CEOs demand that DevOps and security teams prioritize accelerating business processes. 

    Even when organizations race to release updates despite security consequences, the threat landscape is becoming more complex, faster and automated. In the third quarter of 2018, FortiGuard Labs noted almost 34,000 new malware variants—a 40% increase over the second quarter and a 126% increase over the first quarter. Cyber criminals now use advanced technologies such as artificial intelligence (AI) and swarm technology—as well as DevOps itself—to create single-use malware that targets a particular organization across several points of the attack surface. 

    Thus, companies have realized that security must be woven into the DevOps approach–hence the birth and rapidly increasing adoption– of DevSecOps.

    The Security Skills Gap

    At the same time that organizations are embracing DevSecOps, the paucity of skilled IT talent persists. A report from 451 Research on the IT skills shortage notes organizations are facing talent gaps across a range of IT specialties, including database administration (31%), general network administration (36%) and server/systems administration (43%). 

    The dearth of seasoned cybersecurity professionals looms even larger, with an estimated skills gap of just under 3 million individuals. Understandably, 53% of respondents to a report from ESG admitted to a significant shortfall of cybersecurity skills at their organization. These figures represent the general cybersecurity personnel needed to support and secure traditional network environments, to say nothing of DevOps.

    Overcoming the Hidden Challenges

    In DevOps, any security implementation that hinders speed will be seen as a threat to their primary objectives. Traditional IT teams and DevOps teams are often at odds here. IT will suggest the use of security tools that DevOps sees as causing bottlenecks, which runs counter to their primary objectives. However, while DevOps may be highly proficient at building applications, they often lack the expertise and skills to do so securely.

    An answer to this challenge is to add a cybersecurity specialist to each DevOps team to create a DevSecOps team. This DevOps security specialist (or team of specialists) can guide application developers through the shared responsibility model, helping them stay on track with both development and security requirements. They’re also there to provide strategies for consistent security policies across and between all their cloud workloads and services, all while protecting the DevOps mission of reliability and high performance.

    With DevSecOps in place, the team can choose, implement and manage tools that will better equip them to meet the goals of speed and security. Take, for instance, the use of security solutions that are offered as a service (SaaS) or web application firewalls, which can auto-scale. That allows publicly facing web apps to grow as needed, without compromising security. The right tools can also be eased into deployment with minimal effort. Some even have built-in functions that cover security during deployment, maintenance, scaling and for all the fine-tuning that needs to take place throughout ongoing use and development.

    Once an organization has augmented DevOps with DevSecOps, teams can integrate security from day one of each new project. DevSecOps team can also develop the necessary cloud security playbooks and ensure that those guidelines are followed. DevSecOps can even directly affect the bottom line when they help prevent violations against regulatory requirements and the fees and penalties that come with them.

    Closing the Gap

    The fact remains, though, that good skills are hard to find. This makes it harder to have a DevSecOps team in place. Applications being built in or migrated to the cloud need to be protected against new threats that propagate across workloads—and from cloud platform misconfigurations at the user interface and application programming interface levels. 

    Addressing this challenge requires a concerted effort on the part of both the private and public sectors to develop just-in-time training and education programs to develop cybersecurity skills that are critical for every organization, and furthermore educate for cloud security and DevSecOps practices. More and more organizations are in need of training programs that focus on the cloud security skills necessary to upskill their current and future workforce.  

    Failure to understand and implement a security strategy can make an organization susceptible to policy and enforcement gaps, as well as risk to business continuity and digital operations, which can determine whether or not your organization is able to thrive in today’s digital landscape.

    Proactive DevSecOps

    As organizations evolve from DevOps to adding a DevSecOps practice, the talent shortage continues to loom large. This shortage could pose significant security risks to your organization. But, you can play an active role in bridging the skills gap by identifying training programs and certifications available for your teams focused on security. Employees completing these trainings and earning relevant certification can then work with IT security and DevOps teams to provide the security that enables the speed you need to remain competitive.

    — Lior Cohen