devopsy.kinsta.cloud

Category: DevSecOps

  • Playwright: A Modern, Open Source Approach to End-To-End Testing

    Playwright: A Modern, Open Source Approach to End-To-End Testing

    I was excited when I started writing my first end-to-end tests years ago. The idea was promising; create an automated test suite that spins up a browser and mimics your user’s behavior. Test your website end-to-end. Frontend, backend, your database—you name it. All these automated tests would cover everything! Unfortunately, it wasn’t all roses and sunshine. The implementation was more complicated than expected and the results were not as stable as I had hoped. On one hand, I expected that learning a new testing framework would take time. But on the other hand, investing time and money just to end up with flaky tests and countless false positive alerts was not acceptable.

    Luckily, testing frameworks evolved, and new libraries make end-to-end testing more approachable and, most importantly, less flaky. One of the more recent solutions is Playwright. Let me tell you more about it!

    Write Your Test Code in Four Different Languages

    While Playwright’s default choice is Node.js, it offers support for three other languages: Java, .NET and Python. You’re in good hands if your team uses one of these languages because no one wants to learn a new language just to write some tests, right?

    Cross-Browser Tests on the Back of the Chrome DevTools Protocol

    Contrary to other solutions, Playwright doesn’t use the WebDriver protocol. It leverages the Chrome DevTools protocol to communicate with Chromium browsers (Chrome/Edge) directly. This approach allows for more direct and quicker communication. Your tests will be more powerful and less flaky.

    But Playwright doesn’t stop at Chromium browsers. The team behind the project understood that cross-browser tests are essential for an end-to-end testing solution. They’re heavily invested in providing a seamless experience for Safari and Firefox, as well, and even Android WebView compatibility is in the works.

    Testing your sites in Chrome, Edge, Firefox and Safari is only a configuration matter. And this saves time and headaches!

    Fast Tests Due to Auto-Waiting

    It’s not only about automating multiple browsers, though. If your tests are hard to write because you have to place countless “sleep” statements everywhere, your test suite will take hours to complete and become a burden.

    To avoid unnecessary waits, Playwright comes with “auto-waiting.” The idea is simple: Instead of figuring out when a button is clickable by yourself, Playwright performs actionability tests for you.

    Let’s look at an example:

    “`
    await page.click(‘[data-test=cta]’)
    “`

    There’s nothing fancy about clicking a button, right? Except in Playwright, there is. This innocent “click” call includes additional convenience functionality. When you call “click,” the framework checks and waits for a matching DOM element to be:

    – attached to the DOM
    – visible
    – stable (it’s not animating or moving)
    – able to receive events (not hidden or obscured by other elements)
    – enabled

    All these checks guarantee that your target element is ready to roll when you want to interact with it and, more importantly, it’s clicked as quickly as possible. Don’t worry about DOM elements. Just interact with them and let Playwright figure out the rest—no more wait statements!

    Auto-waiting is built into all of Playwright’s interaction methods so that you don’t slow down your test suite. Your tests will be easier to write and run as quickly as possible.

    But that’s not all! While I use Playwright via the command line, it also comes with handy tools to make end-to-end testing easier.

    Valuable Tools to Create, Debug and Analyze

    Playwright provides additional commands and visual tools to make the writing of end-to-end tests as accessible as possible.

    Record Scripts With Codegen

    To give you a head start with your first test, use Playwright’s test generator.

    “`
    npx playwright codegen playwright.dev
    “`

    Only a single CLI command is necessary to spin up two new windows: An interactive browser and the Playwright Inspector. The fun part is that every interaction in the browser window will be recorded and shown in the inspector.

    If you want to create a Playwright test that navigates somewhere, clicks a button and fills out a form, these tools are perfect for creating an initial setup.

    Playwright library

    Replay the generated script, adjust your selectors and assertions, and you’re good to go!

    Congratulations! You just created your first end-to-end test, and I bet it didn’t take you more than a few minutes to get there.

    Debug Your Scripts While Running Them

    When your test suite grows, you’ll find yourself in situations where you must debug complex scripts. For these cases, Playwright provides a handy debug view.

    Enable debug mode by explicitly calling `test –debug` or defining the environment variable `PWDEBUG=1`.

    “`
    npx playwright test –debug
    PWDEBUG=1 npm run test
    “`

    Playwright then opens a browser and the inspector walks you through your test case, similar to your editor’s debugger. Additionally, it highlights the touched elements with a little red circle so you can see what was clicked or filled.

    Playwright debugger

    This approach is perfect for debugging tests during local development. But what about debugging failed tests that run in CI or on a different machine? There’s a solution for that, too!

    Analyze and Debug Previously Failed Tests

    In an ideal scenario, you’ll run your tests automatically on commit or deployment in your CI/CD pipeline. But debugging failed tests from a different environment can be a massive challenge.

    Luckily, the Playwright framework allows you to enable a trace mode. Trace mode records all script actions while screenshotting and keeping the website’s state.

    Trace view

    Then, if your tests fail on your remote servers, use the provided Playwright tooling to load a generated `trace.zip` file and inspect the test to find out what failed. Method calls, a timeline and screenshots will be available—it’s time travel debugging at your service!

    A Refreshing Approach to Synthetic Monitoring and Testing

    These are only a few things that make Playwright my testing framework of choice today and the team behind it develops fast and ships new releases every month.

    If you want to learn more about the Playwright framework or to chat with the maintainers, there’s a very active community Slack channel with many friendly people.

    So, do you still avoid end-to-end tests because they used to be slow and flaky? I hope that I could convince you to give it another try. Because there’s nothing better than knowing that you didn’t break things while developing and shipping software.

  • Federal Agencies Share DevSecOps Guidelines

    Federal Agencies Share DevSecOps Guidelines

    The National Security Agency (NSA), Cybersecurity and Infrastructure Security Agency (CISA), and the Office of the Director of National Intelligence (ODNI) have published a set of DevSecOps best practices based on the Enduring Security Framework (ESF).

    Developed by a public-private cross-sector working group led by NSA and CISA, the ESF framework was created in the wake of the breach of the SolarWinds software supply chain that led to malware being distributed downstream to a number of organizations that relied on the company’s IT management platform.

    The Securing the Software Supply Chain for Developers set of best practices published by NSA, CISA and ODNI takes the ESF a step further. The guidance walks organizations through the processes required to develop secure code, verify third-party components, harden build environments and deliver code.

    The best practices provided are the latest dividend of an executive order issued by the Biden administration that requires every federal agency to review the security of its software supply chains. The expectation is that enterprise IT organizations will conduct similar reviews using the best practices and frameworks defined by the federal agencies that specialize in cybersecurity.

    The challenge, of course, is embedding those best practices within existing DevOps workflows. In theory, more responsibility for cybersecurity is shifting left toward application development teams. However, the level of cybersecurity expertise that exists today among application developers is limited at best. Cybersecurity was an elective in most developers’ training and education programs, so it should come as no surprise that few of them ever attended a cybersecurity class.

    There’s a clear need to improve cybersecurity training for developers, but the real work involves automating cybersecurity processes within the workflows managed by the tools developers employ to write code and the platforms that DevOps teams use to deploy applications, said Mitch Ashley, principal analyst for Techstrong Research, an arm of Techstrong Group that is also the parent company of devopsy.kinsta.cloud. “Security needs to be part of the toolchain, workflows and how the software is designed,” he said.

    Ideally, organizations should appoint a security champion that will work with development teams to consistently implement a set of DevSecOps best practices, added Ashley.

    Of course, the security champion doesn’t eliminate the need for cybersecurity professionals. Rather, it reduces the pressure when there is a chronic shortage of cybersecurity professionals. In the absence of any formal cybersecurity review during the application development process, it’s simply too easy for developers to inadvertently introduce malware into a software component or misconfigure a cloud infrastructure service. In either case, cybercriminals are increasingly finding it easier to compromise software supply chains at multiple points.

    It’s not clear how much the federal government’s guidance might impact the overall state of application security. However, one way or another the level of security scrutiny being applied to how applications are built and delivered is only going to increase in the months ahead. The challenge is finding a way to deliver secure code without necessarily slowing down the rate at which applications are currently being built and deployed.

  • Supply Chain Security: Has the Next SolarWinds Already Happened?

    Supply Chain Security: Has the Next SolarWinds Already Happened?

    More than two years after the now-infamous hack of the firm SolarWinds, the incident is very much in the foreground of conversations about cybersecurity defense, threat detection and incident response. As evidence of that: The SolarWinds incident was the subject of five separate talks and panels at the recent RSA conference in San Francisco. Dozens more talks addressed issues related to supply chain security that–directly or indirectly–invoked the SolarWinds incident. 

    That’s because the SolarWinds attack raises important—if not existential—questions for both software development and DevOps teams and the companies running the software. First: If sophisticated cybercriminals can insert malicious backdoors directly into application code from reputable developers, what organization can afford to blindly trust software? Second: If it is now the responsibility of software teams  to spot sophisticated, methodical compromises of their development pipelines, what is the best and most reliable means of doing that? Finally, if it is also now the job of software consumers to monitor for—and detect—compromised binaries from otherwise reputable vendors (like SolarWinds), how should they go about doing that given ongoing staffing and technology constraints? 

    The answers to these questions are matters of debate within DevOps, software development and information security communities. What is beyond debate is that the SolarWinds incident was not a fluke. In the months since that attack, attackers—both criminal and nation-state-backed—have leveraged software supply chain security blindspots to target development organizations as means of gaining access to sensitive IT environments. While we haven’t yet heard of another incident of the scale or importance of SolarWinds, it is entirely reasonable to assume that the next SolarWinds-caliber compromise could already be unfolding, but has yet to be discovered and/or publicly acknowledged. 

    Software Supply Chain Security Weaknesses

    SunBurst, the malicious implant in SolarWinds software, endures as a topic of interest and conversation for information security professionals and for DevOps and development organizations precisely because of the ways it exposed glaring weaknesses in the security of development organizations and software supply chains. As noted in my analysis of the hack in December 2020 and FireEye’s analysis of the same compromise, the SolarWinds hackers gained access to the company’s development environment months before they actually launched their attack. During that time, they closely studied the underlying source code and mimicked SolarWinds style and nomenclature as they added malicious code and integrated it into the SolarWinds Orion application via a compromise of the company’s build system.  

    Though carefully planned and conducted, the SolarWinds attack wasn’t foolproof. The attackers left clues, such as their additions of native Windows functions, use of cryptography, compression and more. The attackers obfuscated these using compression and Base64 encoding designed to fool simple threat hunting rules that looked for their plaintext variants and other suspicious strings. Still, the obfuscation repeated throughout the implanted code was enough to send up a red flag to anyone who bothered to look closely. That left the attack open to discovery. 

    The problem is this: Very few software publishers have the staff, expertise or tooling to comb through massive codebases to spot a handful of sketchy obfuscated strings or analyze large binaries for deviant behaviors before deploying them. As evidence of this, ReversingLabs sponsored a survey of more than 300 IT professionals at software-driven enterprises. It found that 87% were aware that software tampering, like what occurred in the SolarWinds attack, could lead to enterprise security breaches. About 40% of those surveyed said CI/CD toolchain exposures posed a risk to their organization. Still, barely a third of respondents (37%) said they had the means to detect such software-tampering attacks. 

    Threats to Software Development Organizations Grow

    Attackers are aware of this weakness within the software development pipeline, which is why threats to development organizations are metastasizing. Just in the last year alone, we have seen a series of attacks on development infrastructure and software supply chains. In one April 2021 incident, attackers were successful in pushing malicious, unreviewed code directly to the PHP main branch, ultimately resulting in a compromised, formal version of PHP being pushed to all PHP websites. That same month, there was a breach at the firm CodeCov, which involved unauthorized access to and modifications of the company’s Bash Uploader script. An investigation of that incident ultimately concluded that the alteration of a single line of code in that script by an unauthorized third party enabled the attackers to (potentially) export information stored in the users’ continuous integration (CI) environments. 

    More recently, ReversingLabs observed proof-of-concept and “white hat” activities testing so-called “dependency confusion” attacks, in which attackers set up malicious packages in public repositories like GitHub that squat on the names of private, non-public repositories and sport high version numbers. Misconfigurations in package management systems can result in these malicious components being imported in lieu of the legitimate, privately managed namesakes. As ReversingLabs found, a number of malicious modules discovered on the npm platform were used to target German technology, manufacturing and entertainment firms. It was later revealed to be the work of a white hat firm conducting red team exercises on behalf of those firms. Red team exercises aside, RversingLabs research also detected other, malicious packages in npm, as well as similar attacks targeting Ruby Gems and Python Package Index (PyPi). 

    And that doesn’t even count issues like the Travis CI flaw that exposed (potentially) millions of credentials to developers’ accounts on GitHub, Amazon Web Services and Docker Hub. Lapses like that can fuel supply chain attacks, giving malign actors the ability to insert malicious code into otherwise legitimate open source or proprietary code repositories. After reading such headlines, and without the means to check code intent, how can developers trust their software to update? And how can they do so while at the same time being forced to frequently update their dependencies to reduce or eliminate critical software vulnerabilities?

    Dev Teams Need New Tools

    For software development organizations struggling to meet delivery deadlines and satisfy customer expectations, reorienting to address amorphous software supply chain threats and attacks is a bridge too far. The same is true for enterprises struggling to staff security teams and simply keep on top of targeted phishing attacks or threats to public-facing IT assets. Too often, these entities only find religion about software supply chain attacks after they have occurred. 

    What software development organizations need is a way to secure the CI/CD pipelines and release packages, ensuring that their software builds are free of code tampering and other supply chain risks introduced through build systems. That should include the ability to detect compromises by monitoring for malicious software changes and enforcing security controls and failure conditions on software build processes to ensure that you can release your code with confidence.

    On the customer end, organizations need a way to detect malicious behaviors and unauthorized software changes in compiled binaries before they are released into production, while also monitoring for risks like exposed secrets or exploitable software vulnerabilities that might lurk in compiled code. 

  • The Missing Link in DevOps Cloud Security

    The Missing Link in DevOps Cloud Security

    We’ve all seen the data from the latest Verizon Data Breach Incident Report that shows half of security breaches stem from credential abuse. It’s clear that credential compromise is an epidemic in cybersecurity and can be largely avoided with multifactor authentication and least-privilege policies. Misused credentials have shot up by almost 30% over the last five years, according to Verizon. Engineers and developers are not immune to this plague; they may be even more attractive to attackers because admin privileges allow access that reaches far beyond that of the everyday user. In fact, they are often granted always-on access when, in reality, they only need brief, intermittent access to get their job done. According to Gartner, privileged access carries significant risk. Even with PAM tools in place, the residual risk posed by users with standing privileges remains high. They recommend identity and access management (IAM) leaders implement just-in-time (JIT) strategies and, eventually, reach a posture of zero standing privileges.

    Giving engineers right-sized permissions with only the privileges they need to carry out specific tasks is especially important when these credentials are used to stand up or make changes to production resources.

    Just in Time for Developers

    JIT inventory management has been standard in many industries for decades. Especially in manufacturing, where factories can’t afford to have supplies sitting in the warehouse unused, assembly lines lying idle or have production at a standstill while waiting for parts. In information security, JIT management grants users privileged access for only the amount of time needed for a user to complete a task. This helps minimize the attack surface and avoid the risk caused by long-standing unrevoked privileges.

    In today’s agile environments, development engineers need to move and fix things fast. For example, cloud infrastructure environments are usually set up and maintained by DevOps and development teams via continuous integration/continuous delivery (CI/CD) pipelines. But developers sometimes also need direct, highly privileged access to production environments for more specific tasks.

    While static privileges that allow developers to perform day-to-day tasks are typically provided through the network’s IAM system, organizations often lack the ability to grant and revoke privileges dynamically for cloud environments. That’s because traditional on-premises privileged access management (PAM) and identity governance and administration (IGA) solutions don’t support cloud-native—and increasingly multi-cloud—infrastructures.

    Developers will always need quick, highly privileged access to sensitive cloud environments for activities such as debugging or manual deployment of a service. But those wide-ranging entitlements don’t need to be permanent and can introduce a significant risk if they’re not revoked once the job is done.

    This has left organizations resorting to risky, overprivileged practices. Inactive accounts are the source of many cybersecurity headaches. In fact, that is how cybercriminals breached the network of Colonial Pipeline in last year’s ransomware attack that shut off gas supplies. Overprivileged accounts are another big headache, which was a big factor in the now-famous SolarWinds attack.

    JIT Access

    This is where JIT access comes into play, and can enforce highly granular entitlement management for privileged access in the cloud. Gartner has become an advocate of JIT since it enables a number of security best practices. They recently advised security leaders to implement “a process for quick and easy requesting and granting of additional privileges with minimal disruption to an individual’s workflow.”

    One of the biggest challenges to implementing JIT privilege management is provisioning. Security staff is overtaxed, especially as the adoption of multi-cloud environments keeps adding complexity to their work. Provisioning individual, temporary permissions for each task becomes onerous when there is so much more to oversee.

    JIT privilege management for the cloud has to meet developers’ need for speed, self-sufficiency and transparency. One approach is to use a self-service portal that accepts requests for elevated privileges, manages the approval process, grants authorized permissions and automatically revokes them when they expire.

    A JIT Framework for Developer Security

    This framework provides security and productivity benefits.

    First, it minimizes the cloud attack surface by enforcing temporary and granular least-privilege policies. This prevents attackers that have compromised a user’s credentials from using excessive permissions to access sensitive resources and exfiltrate data.

    Second, it saves engineering teams time and effort by enabling them to quickly submit a request, notify approvers and gain temporary access. This automated workflow eliminates delays and allows DevOps staff to get their work done more rapidly and securely.

    Finally, monitoring user activity during elevated privilege sessions makes it possible to generate reports for all JIT access requests and authorizations to enforce security policies. It also creates an audit trail for forensic analysis in a security incident.

    When implementing a JIT framework for cloud privilege management, consider the following best practices:

    Look for tools that offer developers a seamless experience through a self-service portal. This saves everyone time by automating the submission, processing, approval and logging of access requests.

    Define owners for each step of JIT approvals including who manages eligibility, who approves the scope of access and who reviews requests.

    For low-risk access requests or those involving non-production environments, use policies that eliminate the need for human intervention. The system should be able to log requests, assess the stated reason for the escalated privileges and the timeframe and respond rapidly.

    Reducing the Attack Surface With JIT

    Giving DevOps staff unfettered access to sensitive resources, especially in production environments, for extended periods of time violates basic security hygiene best practices. JIT privilege management enables organizations to keep engineering activities on track and on time without incurring excessive risk. By reducing the attack surface associated with DevOps identities, even if a user’s credentials are compromised, a hacker will not have the keys to the proverbial kingdom. This will limit the blast radius of an attack.

    The challenge companies face is extending traditional privilege management capabilities to cloud environments. Fortunately, new cloud-native advances in IAM and PAM functionality are making it possible to close this missing link in cloud security.

  • Why DevOps Teams Need Security Engineers

    Why DevOps Teams Need Security Engineers

    In the episode of View with Vizard, Mike Vizard sits down with Om Vyas, chief product officer for oak9, as he explains why security engineers need to become part of every DevOps team. The video is below followed by a transcript of the conversation.

    Vizard: Hey, guys. Thanks for the throw. We’re here with Om Vyas, who’s Chief Product Officer for oak9. We’re gonna be talking about all things related to DevSecOps and cloud security. Om, welcome to the show.

    Om Vyas: Yeah. Thanks for having me, Mike. 

    Vizard: We, of course, have been talking about DevOps for the better part of a decade. And then we decided that the security people couldn’t be left out, but they wanted to have the word “sec” thrown in there to make things a little more confusing, so then we got DevSecOps. And some of them even argue that I guess it should be SecDevOps. But who knows? My question to you is it seems like we’re still struggling with getting this idea across to people. It’s become more important in the cloud era, obviously, ’cause more developers are programming stuff. But what, from your perspective, is the fundamental challenges that we’re seeing with getting people to really implement DevSecOps practices?

    Vyas: Yeah. I think if we take a quick step back, I think you hit it on the head. There’s DevOps, there’s DevSecOps, there’s SecOps. There’s a lotta buzzwords and a lotta different ways that these adoptions are being thrown around. I think fundamentally, my take is DevSecOps or DevOps, it’s first and foremost a cultural transformation. The technology and the people process really come after. So really, one of the biggest challenges is that embracing this notion of being able to work together, being able to have shared responsibility and a collaborative environment.

    If you think about kinda where we’ve come from, and in fear of dating myself, I’ll say that when I used to develop long ago with my DevOps, my development teams, our focus was only towards application development. And then it was like the packaging, the art effects, once they were created, it was like, “Hey, kinda ” the notion around, “Hey, throw it over the wall to IT ops people,” or having formalized security review processes, right? And really, this cultural transformation is, I think a key area where I think a lotta the folks in the organizations, they’re struggling. They’re struggling to embrace the culture of DevOps, culture of DevSecOps.

    And then when you factor in, as you said it yourself in the beginning, that the security people are getting left behind. And what that really means is they don’t have the visibility and they don’t have the automation to be able to keep up with the amount of churn that’s happening with the development, the DevOps. Because the dev teams are using the automation, which is the technology part of this transformation, to really create this velocity in which they are churning over new features, doing rapid deployment into production. Five, seven years ago, we used to do maybe one release a month, one release a quarter. And now, I’m seeing organizations, ours included, where we’re doing tens of releases a day, hundreds of releases a day. So this is really where the security or the “sec” part of the DevSecOps is really struggling to keep up with that velocity and that change.

    Vizard: Does that mean we need to slow down? Or can we build and deploy secure applications just as fast? What’s the tradeoff?

    Vyas: Yeah. I mean, I think maybe slow down is not the right answer. Because I think velocity is really being utilized as a differentiation for a lotta the – lotta the organizations. I think the right answer is – I think, not necessary just slow down to a halt. The right answer in my opinion is really, how do we integration security into the processes of DevOps to truly give them the visibility at the point of time when these changes are happening and then really arm them with the automation that they really need to give them full visibility, give them ability to be able to really quickly identify how that application is changing? How are those new capabilities that are being added, and how is that impacting security of that application? And the sooner we give visibility to security engineering of all of those changes and where security matters in those changes, the sooner we can continue to operate with that velocity.

    Vizard: How do we go about doing that? ‘Cause a lotta the developers, they didn’t really set out to build something that’s insecure. It’s just more often than not, they have a tool that lets them manage a cloud as infrastructure, as code. But then they misconfigure something ’cause they’re not security experts, and maybe a port gets left open or something like that occurs. So how do we lock all that down and how do we kinda tighten that up without necessarily requiring the developer to become a security expert, ’cause that seems unlikely?

    Vyas: Yeah, you’re absolutely right. I think that a developer is not a security expert. And I think if you look at kind of where dev engineering is today, especially with the Cloud Native architecture, especially with DevOps, it’s dev engineering or developers are taking on lot more responsibility, right? But I do think as a developer myself, what I’m interested in is not that I don’t care about security. It’s that what I wanna know is, “Hey, help me understand what I need to do to move fast. What do I need to do? What are the five things or ten things or whatever number of things that are related to security that I can do to not only help my security team, but really quickly move this feature capability from development and implementation to testing and to production?”

    So from that perspective, I think what really also helps is if I have the ability to, within my own workflow, within my own tools – if somebody educated me and gave me kinda the prescriptive guidance of, “Hey, here is all you need to do to ensure that all of these different security aspects of security-related things in this infrastructure’s code that you change. Here’s what you need to be aware of. Here’s what you need to change, and to really help you move forward.” I think that’s what, I think as a developer, I’m looking for. Because again, I have many, many different things that I’m managing today beyond just application code. 

    Vizard: Mm-hmm. How automated can all this get? Seems to me the key to this whole thing is if the developers still need to program the cloud environment, we need to find a way for them to make fewer mistakes, put more guardrails in place, and kinda just gently lock down the environment, I guess, is the phrase I’d say. So what is the challenge, what is the state of the art at the moment?

    Vyas: Yeah. I think it’s interesting. You said “gently lock down the environment.” I think absolutely, right? I mean, I think that developers, what we’re seeing today with a lot of our customers is they want lotta freedom because that freedom will enable them the agility and the velocity that they are looking for. But at the same time, we do need guardrails because they’re important.

    And I think how do we gently lock down the environments? I think that’s kinda the key. I think one way we can talk about this is that when we think about that infrastructure’s code, we’re really talking about codifying the entire environment under which your application is running in the cloud. And what that actually enables us to do is be able to then say, how do we actually remediate? How do we actually focus on automation that doesn’t always just point out all the challenges or all the problems in that code around security? But then taking it that last mile, as I like to think about it, which is actually fixing it for that developer.

    So I think that’s really where the trends are. And I think that’s really where things need to be, is really say, “Hey, I analyzed this infrastructure’s code. I have identified where all of the security challenges are. Here’s how to risk-appropriately secure this code. Because all of this infrastructure’s code is declarative. So it’s predictable in how it can be changed and what the impacts would be.”

    So I think that’s really where it needs to be, is when I look at some of our customers and infrastructure’s code, it’s getting pretty complex, right? So we have customers that have 40,000, 50,000, 60,000 lines of Terraform code. So even cloud infrastructure code or infrastructure code itself is getting pretty complex. So no security engineers or no number of security engineers or even DevOps or developers can actually keep up with understanding, what are all the security implications in this code? So the real answer in my mind is, how do we automate this thing, take it to the last mile, and say, “Hey, here are all the recommended changes in this code as far as the security aspects of it is concerned?” So as a developer, now I can focus on the functional aspects, and I have an automated tool, an automated process, that will really enable to me to then not have to worry about the security aspects of that infrastructure’s code.

    Vizard: And maybe no one ever needs to know how much I don’t know about security. But do you think that we’re gonna see a lot more of so-called artificial intelligence thrown at this problem? And is that part of this automation curve? Or are we making too much of AI and security these days and it’s just gonna require something a little more fundamental?

    Vyas: Yeah. I think the short answer is yes. But I think my opinion is that AI kinda gets thrown around a lot. And I think when we’re talking about some of the deeper learning concepts in AI, we’re talking about some neural network things, I think those things are coming. But I think we still have to recognize that we’re in the infancy stages of this. So I think really to me, it starts with the fundamentals first. Can we actually provide these recommendations, this automation, to really help educate the user, that developer, that practitioner, that is today, lot more often than not is kind of utilizing lotta the modularized code that’s available in the web without really understanding, what are the implications as it relates to their own world or their own applications or their own context?

    And lemme sort of dive in deeper. So not all applications or workloads in the cloud are created equal. So if you are a health care company that’s securing health care applications that’s utilized by your patients, that’s using PII data, their requirements around security may be completely different than if I’m building a marketing website where most of the data that’s on that website is publicly available. Both of those applications are not created equal. So how do we secure those two different applications is actually different.

    So there’s an element of this dynamic nature. There’s element of understanding, what are the requirements of that application? What are maybe the compliance adherence to those applications? So really then, when we talk about securing those two applications, that’s also different. So it’s really what I think about the context behind that application and how you secure that application really becomes important or a focal point of, then what are the recommendations that you’re providing to that end user, that developer, within that infrastructure’s code? So I think that has to take that into consideration.

    And then I think, to your point, it’s not simply just saying, “Hey, here are the ten things you need to do, developer. Go and here’s a tool that will actually remediate it for you.” That’s great. But I think part of it can be used really as an educational tool to really better understand, “Hey, developer, help us a little bit. Help us understand the context of this application. What are, what is it that you’re trying to do?” And then I think once you kind of cover those fundamentals, then I think as we go deeper beyond the infancy stages of this cloud infrastructure journey, then I think what we can do is we can learn behaviors. We can learn from the industry that this developer or this organization is in. And we can say, “Yeah, this is a health care industry vertical or this is a particular type of architecture or workload that’s being deployed in the cloud,” and secured.

    And then I think we can start to get into intelligence around deep learning and other heuristics to say, “Hey, how do we react to then risk-appropriately secure this application,” right? Because otherwise, if we start treating all applications, all workloads that are run in the cloud, in the same way, there’s a lot more that we’re asking of the developer. There might be hundreds of things that we need to do to risk-appropriately secure a health care application that’s using PII data versus we might only need to do a handful of things to ensure that a marketing website that has publicly-available data is secure.

    Vizard: What will be the role of the security team in all of this? Because one of the things that comes to mind is if we are shifting more responsibility or at least the enablement left, then what exactly does the security team need to do to kinda work alongside, then? Or will they just kinda set the policies and the rest of it will be automated?

    Vyas: Yeah. I think, first and foremost, about kinda the role of security engineering today in the DevOps world or DevSecOps world, is really understanding and gaining visibility into what’s happening. Because if you have engineering teams that are deploying tens of times a day into production, they’re fundamentally changing, potentially, the architecture of that entire application. So I’ll give you an example. Today, with lotta the sort of building blocks or the way I like to think about it is cloud service providers are really offering these Lego pieces. So they’re outta-the-box capabilities.

    Now, I can plug in an analytics service into my architecture and seamlessly provide analytics dashboard into my application or into my workload that I didn’t have a week ago. But that fundamentally changes, or it fundamentally has the ability to change, the architecture, and therefore the security implications. So first and foremost from the role of security engineering is really getting the visibility and getting the visibility as early as possible so that they can understand how, fundamentally, this application is changing and what are the security implications of that. So it, and I’ll say for that aspect of it, this is the people process side of embracing DevOps, is really engaging that security engineer as early as having that requirement or having that capability from business that says, “Hey, we need this analytics dashboard,” really getting the security engineering involved as early as possible.

    And from that point forward, again, then it goes back to technology and automation. How can security engineering really understand, what are the implications of adding this analytics service from AWS or some analytics service from GCP or Azure? What does that mean for the security of my architecture? So there might be hundreds of different configurations. There might be thousands of different implications around putting in that one analytics service that’s available outta the box as a feature or service from the cloud service provide. And the reality is no security engineering can actually keep up with that.

    So having a tool that really understands down to the configuration level, what are the impacts or the configurations of that analytic service to securely deploy that service and then what security engineering can do is security engineering can say, “Hey, let me understand the implications and the architecture when my development team is adding this analytics service.” And if we start early, if we get the security engineering involved early, if we enable, then, automation to take away the implications of down to the hundreds of configurations of securely deploying that analytics service away from security engineering, but then give them the visibility, then I think we have a really great starting point to enable all of the teams to work together with the same operational velocity that the dev engineering really desires.

    Vizard: Mm-hmm. What is your best advice to folks? I mean, should I just take all these developers and security people and lock ’em in a room till somebody comes to an agreement? Or is there some more intelligent way to go about this?

    Vyas: Yeah. I think locking everybody up in a room is not the right answer, right? I mean, I think what I’ve seen work really well is there’s lotta different ways to sort of come to a solution. I think there’s no one silver bullet here. But I think depending on the size of the organization, depending on the size of the teams, I think one of the things that should be done, and I’m starting to see some trending around this, is really add a security engineer to the engineering team as a whole so you really have a dedicated security engineer that’s spending part of their time with these engineering teams.

    So typically, we’ve seen, in more of the modern development shops, you see lotta the dev engineering, it started out with just developers in a single team. Then we kinda focused on scrum teams. So there was development teams, there was a product engineering or QA involve.We’ve seen a DevOps engineer or SREs get involved in these teams. And I think, to me, the answer is, again, being able to put that security engineer in these teams, as well, to work closely with the developers, work closely with the SREs. Because then, what will start to happen is that engagement, that early visibility and understanding. So I think that’s one.

    And then I think the other area for lotta the larger organizations where this may not work is I think the notion around excellence centers , so you have visibility across these little center of excellence teams. So there’s cloud ops teams, there’s security teams. It’s really good to incorporate lot of your security engineering and security architects into these cloud center of excellence teams. Because I think then again, it goes back to that visibility, goes back to engagement, and it goes back to helping security teams really understand, what are the different architectures? What are the different capabilities that lotta the dev engineering groups are using today? Is it the serverless architectures? Is it the micro services that are being built with containerized architectures?

    And then what security engineering can start to do is they can start to build these reference architectures or guardrails around 80 percent of the, “Hey – ” recognizing that 80 percent of the workloads across my organizations are built with serverless architectures, as an example, in the cloud, or it might be that containerized architectures in the cloud. And that will really help them start to build these guardrails in an effective way. And then I think, you talked about sort of, early on, gently locking down environments. So I think if we then start to think about, in the DevOps pipeline, how do we incorporate these security automation tools that enable us, both the development teams, the SRE teams, and the security teams, to identify, what are all of the security underpinnings, vulnerabilities, design gaps that we are identifying, be it with the application code, be it with runtime, be it with cloud infrastructure security?

    So it’s really, where possible, building these automated tools within the pipelines without actually blocking the pipeline so that you’re actually what you’re doing is you’re really enabling the development engineering and the DevOps engineering to really start to build in the process around, how do we ensure that all of these security design and the best practices are actually incorporated within the pipeline? And then as the teams learn and grow, then these tools can become those guardrails. So maybe you’re not blocking or completely blocking the pipeline or failing the pipeline at or in the development or sandbox stages, but then you start to guardrail those as that feature capability’s moving through user acceptance testing or staging or before it goes into production. And I think those are some of the key things that can be done to really enable more of this collaborative, but shared responsibility culture that I think is really going to drive lotta the benefits that are needed to build security in.

    Vizard: Hey, Om, thanks for being on the show. It’s all about making sure that if you find a pattern, build a guardrail, and rinse and repeat.

    Vyas: Yeah, absolutely. Thanks for having me, Mike.

    Vizard: All right. Back to you guys in the studio.

  • Avoiding Security Review Delays

    Avoiding Security Review Delays

    In the summer of 2021, I had lunch with a senior security developer at one of Seattle’s leading tech firms. Even though we were relaxed in the sunny and cool afternoon of the Pacific Northwest, there was no doubt my friend was frustrated. Part of his job was running code through a popular scanning product and then evaluating the results. It was a tough role because he sat outside the team that was developing the code and he was often under pressure to run his analysis within the constraints of their deadlines. 

    All too often, he had to deliver the bad news that the scan had found security issues. But that alone wasn’t why he was frustrated. His issue was that the tool that he was using was taking several hours to complete scans. He’d taken to starting the scan at the end of the week and hoping that it would be complete by Monday morning. The very tool that was supposed to make his job easier was actually a headache.

    This is one of the fundamental challenges of analyzing software systems for security vulnerabilities. All too often, developers and stakeholders focus their project timelines on the functional specifications. Teams work feverishly to rapidly iterate from prototype to finished product. 

    Security concerns are never disregarded, but the culture of “Let’s see if we can get it working the way we want first and button it up later” ends up costing more in the long run because the evaluating and testing cycles take longer. Not to mention that if substantial security problems are found, it can be difficult to remediate them without breaking things that were supposed to already have been completed. This problem is becoming acute in the SaaS world where solutions are being built faster than ever before with little to no security oversight.

    The Right Tools and Software Are Critical

    First, managers and executives need to mandate that the entire software development lifecycle and every environment should have security. Particularly for web-based SaaS tools, any vulnerability that exists is a potential attack vector. Attackers know that going after a dev version might not yield a jackpot, but it might give them access that exposes credentials that work in later versions or identifies a vulnerability that flies under the radar into production.

    Second, automation and tooling are simply the name of the game. Using tools with code repositories, pipeline management and testing frameworks help ensure the elimination of as many potential vectors of error as possible. Security checks should be the same. We can’t always rely on the skills of a developer or an administrator to follow security best practices. 

    We need tools that provide reliable and rigorous checks. Moreover, these tools need to run frequently. SAST scanning should be fast enough that a developer can run it with every save, or at least every commit. Coupling SAST and IAST scanning together is the gold standard for secure development. Every time code is compiled into a dev or test environment, it should be tested to make sure that vulnerabilities haven’t crept in and that the permissions and architecture of the system don’t work at cross purposes to expose vulnerabilities.

    Finally, there is a fundamental issue that must be addressed in every manager’s office. Code quality is not the same as security. Sure, there might be some overlap, but it doesn’t make sense to have 200 rules for code quality and 50 for security. The two things are not equal, and it’s a fallacy to expect that having high-quality code without frequent security checks will ultimately lead to less time spent in security review. Don’t miss the forest for the trees. Make sure security is the team’s top priority.

    Advocating for Security in SaaS

    My friend didn’t have a problem holding teams accountable for the code they’d written. In fact, he relished being able to insist on high standards and was proud that his organization empowered him to do so. That’s exactly what we need in the software industry: Individuals who are unafraid to demand the best from their peers. But it’s imperative that we provide effective tools they can rely on to succeed.

    The problem of security testing delays are not new to software development. It has been well-documented that it can take 640x the money to fix bugs in production, but due to a lack of effective tools this problem has been amplified in SaaS development. A SaaS development environment presents unique challenges that cannot be met by just employing code analysis or configuration testing for SSPM vendors. It requires a holistic view of the environment and the ability to use customization to be effective in the identification and remediation of vulnerabilities. 

    With a comprehensive view, those responsible for security and those implementing it will see security as a necessity to innovate quickly rather than a necessary evil that is put off until last in hopes it can be avoided. Maybe then my friend won’t be as frustrated next time we meet.

  • Cycode Expands Scope of AppDev Security Platform

    Cycode Expands Scope of AppDev Security Platform

    At the Black Hat USA 2022 conference, Cycode this week announced it has added static application security testing (SAST) and container scanning capabilities to its software composition analysis (SCA) platform that is based on a graph database.

    Amnon Even-Zohar, director of product management for Cycode, said the addition of these tools brings to eight the total number of modules that are now integrated with its Knowledge Graph database. Those modules address application security requirements spanning everything from infrastructure-as-code misconfigurations and DevOps pipelines to binary and source code analytics in a way that reduces both the number of false positives being surfaced, he added.

    That approach makes it possible to provide more context to software remediation efforts by pulling data from across the entire software development life cycle rather than focusing solely on vulnerabilities found in source code, he noted. Cycode, for example, can pinpoint where vulnerable dependencies are found in test and production environments, said Even-Zohar.

    A Cycode Pipeline Composition Analysis tool also provides insights into which components enable a vulnerability or security issue, how pipeline components relate to each other and if they are present in runtime environments, he added.

    While there’s a lot of interest in implementing DevSecOps best practices in the wake of a series of high-profile AppSec breaches, many organizations still lack the tools required to secure an application development environment on an end-to-end basis. The Cycode platform leverages the data that already exists in DevOps workflows to enable organizations to better prioritize their remediation efforts and lower the total cost of doing so, said Even-Zohar.

    The goal is to not only enable organizations to shift cybersecurity further left toward developers but also further right by more closely integrating cybersecurity and DevOps teams, he noted.

    The DevSecOps challenge that most organizations are trying to navigate is that, historically, a list of vulnerabilities discovered by cybersecurity researchers is shared with DevOps teams without much context. Developers don’t know which vulnerabilities are most critical or, for that matter, whether the vulnerability identified is actually present within their application environment. SCA tools, in theory, could make it easier to identify, for example, where a Log4j or a shell vulnerability might be found. Graph-based tools take that capability to the next level by making it easier to visualize where any instance of a vulnerability might be found. In the absence of such tools, IT organizations would spend weeks looking for each instance of a vulnerability.

    It may be a while before DevSecOps workflows become commonplace but at the very least there is now more awareness of software supply chain security issues. The Biden administration has also issued an executive order that requires federal agencies to review the security of software supply chains. Many enterprise IT organizations are now following suit. One way or another, application security is going to improve. The only thing that remains to be determined is what impact those improvements will have on the rate at which software can be built and deployed.

  • What GitHub’s 2FA Mandate Means for Devs Everywhere

    What GitHub’s 2FA Mandate Means for Devs Everywhere

    Multifactor authentication (MFA) is becoming increasingly standard within software development organizations, with GitHub recently announcing that two-factor authentication (2FA) will be mandatory for all code contributors by the end of 2023.

    This is a smart move. In recent years, bad actors have frequently initiated attacks by accessing source code through the use of stolen developer credentials. Once inside, they quickly penetrate organizations’ entire code repositories and elevate access to company-wide systems. This was the case with the 2021 SolarWinds compromise that ultimately jeopardized federal agencies, corporations and governmental institutions.

    But for some developers with lofty goals and concentrated timelines, MFA is an extra step that has the potential to slow them down. As organizations continue to implement MFA and other foundational security controls, they may face pushback—but they shouldn’t be deterred. MFA is a critical attack barrier that improves individual protections and can contribute to tightened source code security. 

    If organizations approach implementation thoughtfully, they can lay the foundation for subsequent security controls—and build credibility among their users and within the larger development community.

    The Risks of Lax Individual Developer Security

    Organizations’ source code typically powers a host of upstream programs and products, so it’s valuable for bad actors to launch attacks by infiltrating their repositories. Attackers can also hide malware in source code that can be escalated to breach entire systems and third-party organizations. This risk is heightened by the fact that many companies rely solely on antivirus scans and other surface-level monitoring tools to flag source code vulnerabilities instead of combining them with dynamic testing and penetration testing.

    Attackers typically gain access to source code through stolen developer credentials. While single authentication systems—i.e., usernames and passwords—are the easiest to compromise, MFA creates an additional barrier that’s more difficult for attackers to overcome. While this tool does create additional friction in the development process, it’s worth the extra step. Even for hobbyist or open source developers, compromised source code can cause larger repercussions that put other developers at risk. When it comes to cyberattacks, it’s best to assume any entity is a potential target. 

    Beyond individual security, MFA and other foundational controls are critical to safeguarding organizational reputations and adhering to legal privacy standards. By 2023, 65% of the global population will have personal data covered by privacy regulations. Security measures that protect authentication credentials will become even more important in the coming years, especially for organizations producing source code that powers consumer-facing products and programs. With an intentional approach to implementation, MFA can help mitigate potential source code attacks—and form the basis of comprehensive security programs.

    Four Steps to Successful MFA Implementation 

    The benefits of MFA far outweigh the cost of user friction and developer pushback. Get the most out of this authentication technology by following these four steps to implementation.  

    1. Start small and build credibility. Approach implementation gradually to help developers understand MFA’s value. Successful implementation will give your leadership and security teams credibility for future security initiatives, and it will improve processes for security integration into your larger software development life cycle.  
    2. Examine your security program’s value chain. Implementing foundational security controls like MFA provides an opportunity to examine your security program’s entire value chain. Take time to look for potential vulnerabilities beyond your source code repository and flag areas within your development lifecycle that may create risk.
    3. Layer subsequent security controls. With MFA under your belt, build up your security program with other tools and controls—like dynamic and penetration testing, source code scanning, software composition analysis (SCA) and vulnerability management. The more security tools you layer, the harder you make it for attackers to succeed. 
    4. Plan for continuous optimization. Attackers will devise more ways to thwart MFA as more organizations mandate it. For example, the popularity of using SMS as a second form of authentication declined when attackers developed methods to gain control of mobile phones. To keep up with evolving attacker behaviors, continuously optimize your MFA technology and subsequent security controls. 

    MFA Will Become the Industry Standard

    As bad actors evolve attack methods and privacy regulations strengthen, MFA will undoubtedly become an industry standard for protecting source code repositories. 

    To stay ahead of the curve, focus on the concept of MFA throughout your network by ensuring it protects every system entry point. This extra step is an easy price to pay for improved source code protection—and ultimately safeguards every asset, product and program your code powers. Thoughtfully implementing MFA can lay the foundation for comprehensive source code protection.

  • GitHub Brings 2FA to JavaScript Package Manager

    GitHub Brings 2FA to JavaScript Package Manager

    GitHub has made generally available a two-factor authentication tool for the package manager for JavaScript applications maintained by its NPM, Inc. arm.

    In addition, all npm packages have been re-signed and there is now an npm command line interface (CLI) command to audit package integrity.

    Finally, GitHub has added the ability to connect GitHub and Twitter accounts to npm along with a streamlined login and publishing experience with the npm CLI.

    Myles Borins, staff product manager for open source at GitHub, said these enhancements are part of an ongoing effort to enable organizations to better secure their software supply chain by better protecting credentials against being used to take over GitHub accounts.

    There isn’t a huge volume of these types of attacks, but it’s apparent cybercriminals are attempting to compromise the security of software supply chains by gaining illicit access to the platforms used to build applications, noted Borins. Once cybercriminals gain access, they could potentially insert malware in software components that could find its way into any number of downstream applications. That malware can then be used to, for example, potentially launch a ransomware attack or simply exfiltrate data. More troubling still, that malware might begin to spread laterally across an entire application environment.

    GitHub is currently dealing with the fallout from a report that its repositories were attacked; that turned out to be a false alarm. The repositories attacked turned out to be either forks or clones of real, popular repositories with names similar to legitimate repositories on the GitHub platform. No malicious code was actually merged into any of the original repositories; however, it’s apparent cybercriminals are using a technique known as spoofing commit metadata to attempt to trick developers into committing code to a fake repository.

    The challenge that GitHub is trying to navigate is the degree to which some security controls should be enabled by default based on the level of risk, said Borins. Individual developers, for example, might have a higher tolerance for risk than an enterprise IT organization building an application that is core to the processes that drive the business, he noted. In some cases, organizations may opt to restrict access altogether whenever a user name/password can’t be remembered while others will simply limit access to ensure productivity is maintained, added Borins.

    In general, the average enterprise IT organization is likely to have a higher tolerance for disrupting workflows to make sure critical intellectual property is always protected, he added.

    The critical issue, of course, is no one knows for sure what open source component might one day wind up in what application. Regardless of the type of application, developers now routinely reuse code that could contain any number of known vulnerabilities or zero-day vulnerabilities that have yet to be discovered. The foundation for building secure DevSecOps workflows for most organizations at a bare minimum is going to start with implementing two-factor authentication, said Borins.

    Regardless of the DevSecOps approach, just about everyone recognizes there is a need to better secure software supply chains. The one thing that remains to be determined is what impact achieving that goal might actually have on the rate software is being built and delivered.

  • CREST Defines Quality Verification Standard for AppSec Testing

    CREST Defines Quality Verification Standard for AppSec Testing

    At the Black Hat USA 2022 conference, CREST today shared a quality assurance verification standard to improve application security testing. The standard is based on the open source framework defined by the Open Web Application Security Project (OWASP).

    Tom Brennan, executive director for Americas at CREST, said the OWASP Verification Standard (OVS) measures an organization’s ability to execute and deliver assessments related to Level 1 and Level 2 of the OWASP Application Security Verification Standard (ASVS) and OWASP Mobile Application Security Verification Standard (MASVS).

    The goal is to provide a consistent methodology for accrediting companies that provide application security testing services, said Brennan. Ultimately, organizations of all sizes will then be able to require application software providers to pass an OWASP accreditation test rather than testing the security of every application they use, he noted.

    Organizations need to be accredited through CREST Penetration Testing to apply for the OVS program, which requires companies to demonstrate that they can meet the program requirements and execute and deliver Level 1 and Level 2 ASVS and MASVS services. Level 1 determines whether or not applications store or handle sensitive data; that determination then defines whether or not they need the more rigorous controls of Level 2 or 3. Level 1 controls can be checked either automatically using tools or manually without access to source code. Level 2 ensures that security controls are in place, effective and used within the application. In addition, all organizations will need to ensure that their teams have completed CREST’s Skilled Person Register and have each signed the CREST code of conduct.

    CREST is a nonprofit organization that was originally known as the Council of Registered Ethical Security Testers. CREST works with governments, regulators and multinational organizations to improve application security using CREST OVS Mobile and CREST OVS Apps accredited services. It’s becoming more common for organizations to require OVS to better ensure application security in much the same way they require compliance with frameworks such as the System and Organization Controls (SOC) framework, said Brennan.

    OWASP is administered by a separate nonprofit foundation that works to improve the security of software by sponsoring community-led open-source software projects and training application developers. By aligning with OWASP, CREST creates a verification standard to provide a more structured approach to implementing ASVS and MASVS, noted Brennan.

    OVS, of course, is arriving at a time when there is now a much greater focus on software supply chain security in the wake of a series of high-profile breaches. It is apparent cybercriminals have become more adept at discovering and exploiting a wide range of known software vulnerabilities. The Biden administration has even gone so far as to issue an executive order requiring federal agencies to review the security of their software supply chains.

    It may be a while before application testing using OVS becomes routine, but as more organizations embrace DevSecOps best practices to secure their software supply chain it’s clear there is a greater need for application security testing standards. Especially those that organizations can extend versus reinventing the wheel.