devopsy.kinsta.cloud

Category: Continuous Testing

Continuous testing is the process of executing automated tests as part of the software delivery pipeline to obtain immediate feedback on the business risks associated with a software release candidate.

  • Secure Software Summit: Behold the SBOM

    Secure Software Summit: Behold the SBOM

    With supply chain security becoming more of a focus, the SBOM is now viewed as a critical element in shoring up supply chain security. SBOM stands for software bill of materials. At a very elementary level, an SBOM is a list of ingredients. Think about how every food product in the supermarket lists ingredients so, for example, people who are allergic to nuts can avoid nuts. An SBOM does the same thing for software applications. 

    If you want to do business with the U.S. government, SBOMs are no longer optional. Executive order 14028 requires an SBOM for all U.S. federal software procurements. Why the push for SBOMs now? Software companies generally do not list all the components included in the software they produce. Knowing what components are included in applications is essential for proactive vulnerability management. That said, the government mandate for SBOMs only requires a minimal amount of included information—supplier name, the component name and version, dependency relationships, time stamps for versions or modification and information about who authored the SBOM itself. 

    What to Look For in an SBOM

    This minimum required data yields minimal benefit for cybersecurity risk reduction and vulnerability management. SBOMs can actually include many other types of information. What information should you include in SBOMs for software applications produced, purchased or consumed by your organization? Here is a list. 

    Authenticity —Authenticity asks and verifies that someone actually created a listed component. Ideally, this requires a digital signature of some sort. Authenticity can be particularly important in the case of nested components where the supplier of an application includes multiple other components, as is common with JavaScript and Java libraries used for common functionalities. 

    Integrity —This addresses whether a component has been tampered with or modified without authorization. An SBOM should include some measure of integrity. Equally important is proving the integrity of the SBOM itself. Just as CheckSum can tell you whether software you are loading matches what the source of that software indicates, integrity checks should allow you to verify that components on an SBOM (and the SBOM itself) are exactly what you are expecting. 

    Services —An inventory of services called by an application is critical. Modern software does not live in a vacuum. It calls out to the Internet to perform all sorts of tasks and communication with other systems. Those services that extend beyond your application are part of the inventory because they can be an indication of vulnerability or indicate a larger problem.

    Vulnerability Remediation Status —Development teams are constantly balancing the need to remediate vulnerabilities against the risk that remediation will break the application. Some open source projects, for example, have a history of not backporting security fixes. The latest version of the project may break applications. Organizations are forced to choose between breaking an application or taking the risk of introducing a high-severity vulnerability that might be exploited. Responsible organizations that opt to keep the older version and not break their software will backport their own fixes. An SBOM should describe these variations and any other vulnerability remediation activity. 

    Inventory of component security advisories —An SBOM should describe security advisories pending against all components. This is likely to be a dynamic list that you will need to automatically update using one of the accepted vulnerability advisory formats (CSAF and CycloneDX, to name two). In fact, this is one of the strengths of SBOMs—they are designed for automation and to act programmatically. 

    Completeness —It is not always possible to ship complete SBOMs, either for technical or intellectual property reasons. An SBOM should indicate whether it is complete and whether it contains a comprehensive inventory of dependencies and services. 

    Pedigree —This information describes the DNA of the SBOM and the original makeup and changelog of each particular component. An SBOM should describe the original component, exactly what changes were made, who committed the changes and why, what the differences were between different versions and what security issues were fixed with each variation and version. 

    Provenance —This relates to the origin of a component or service. It can describe a repository that the component was downloaded from, a country of origin, the name of a software manufacturer or open source project or other information indicating the component or service’s source of origin. 

    License type —License information is relevant for compliance and to ensure that your organization is not consuming or producing software in a manner that violates licensing terms. Moreso than for security and vulnerability, this information is for risk management, compliance enforcement and auditing. 

    Customizing SBOMs to Meet Your Needs

    The SBOM is a flexible and highly adaptable approach that can be easily customized to fit specific use cases. You can design SBOMs to meet the specific needs of your use case, either as a software provider or consumer. You also may need to modify your SBOM fields and data design as your needs evolve. Fortunately, it is rapidly becoming easier to automatically generate SBOMs using software security tools such as software composition analysis (SCA) and interactive application security testing (IAST) solutions. SBOMs can also be created or read automatically at other points during the build life cycle along the continuous integration/continuous deployment (CI/CD) process. Just as you would not buy a box of cereal that did not list the ingredients, in the future you will not want to consume software that does not have a detailed SBOM. What you don’t know about the software you run can hurt you, and SBOMs will be a crucial tool in providing greater transparency and observability into the software supply chain. 

    About Secure Software Summit
    The Secure Software Summit brought together the world’s leading innovators, practitioners and academics of secure software development to share and teach the latest methods and breakthroughs on secure coding and deployment practices. If you are about developing, releasing and securing software, delivering new features fast and building things right from the start, click on the link below to get access to all of the sessions from this summit.

    https://go.shiftleft.io/secure-software-summit-2022-replay

  • Survey Sees App Testing More Widely Distributed

    Survey Sees App Testing More Widely Distributed

    A survey of 900 application testing professionals conducted by PractiTest, a provider of a platform for managing testing tools, suggested the number of organizations that have dedicated testing teams is declining at a steady rate.

    The survey found a 15% decrease in organizations with dedicated testing teams that have six or more members. At the same time, however, the survey found only 18% of respondents reported that all testing is done by individuals that are not part of a dedicated testing team.

    Joel Montvelisky, test specialist and chief solution architect at PractiTest, said that shift indicates that more organizations are steadily incorporating testing within DevOps workflows. In fact, more than 60% of respondents said DevOps and Agile processes led to teams being more organized and stable, with two-thirds (66%) reporting that development teams were releasing features more frequently with more functionality.

    However, only 38% of respondents said they were active participants in a continuous integration/continuous delivery (CI/CD) workflow. A full 90% said they needed to coach or train other members within the team and just over half (54%) reported that estimations of the time required for development and testing were accurate.

    Two-thirds also said collaboration with developers is also improved when applying DevOps processes, with more than half reporting fewer serious bugs made it into production. More than half also said they monitor software in production and deployment environments. The number of testers who defined themselves as engineers has gone up by 14% year-over-year. However, only 44% report that automation has replaced 50% or more of their manual testing efforts.

    A full 95% or more of survey respondents identified application programming interface (API) testing, Agile methodologies, functional testing and scripting as high on their list of priorities. In contrast, big data testing, internet of things (IoT) testing and machine learning/artificial intelligence (AI) testing were cited as important or very important by only 60% of respondents.
    Finally, about a third of survey respondents noted that non-testers are now involved in testing activities.

    Montvelisky said the PractiTest survey makes it clear that while developers are a little more involved in testing than they were a few years ago, organizations still relied on testing specialists that are becoming more integrated into DevOps workflows. This is especially important as more organizations launch digital business transformation initiatives that require them to build software using the same types of DevOps processes that independent software vendors (ISVs) employ.

    It’s still too early to say just how far testing might shift left toward developers, but as testing becomes more automated—thanks, in part, to increased reliance on AI—the types of tests that will be conducted by a dedicated testing team will evolve. In fact, it’s unlikely automation will eliminate the need for those teams any time soon. Rather, testers will be augmented by AI technologies that will help them to keep pace with the accelerated rate of application development enabled by wider adoption of DevOps best practices within organizations both large and small.

  • Secure Software Summit: Applying Chaos Engineering to Software Security

    Secure Software Summit: Applying Chaos Engineering to Software Security

    Today’s software systems are, essentially, controlled chaos—and lightly controlled chaos, at that. This makes it exceptionally challenging to model the behavior of those systems. Our systems are quickly becoming larger and larger, with more and more moving parts. It is not uncommon for enterprises to have over 1,000 microservices and millions of containers running thousands of applications containing many thousands of open source libraries. 

    Why are things growing more complex? Because from microservices to CI/CD pipelines, from service meshes to blue/green deployments, all these new practices and new technologies help us make our applications and infrastructure more resilient and scalable. The downside is the ever-increasing complexity, which is actually accelerating. 

    In addition, software has officially taken over every layer of the networking and hardware stack (which sounds like an oxymoron, but it’s containers and virtualization turtles all the way down until you hit bare metal). Software is, by definition, easy to rewrite. Thus, the ease of changing software has further increased the number of changes we make to systems. Which adds still more complexity. In fact, software only ever increases in complexity; never decreases in complexity. Software is always trying to do more and encompass more. For that reason, even when you try to simplify software, you just move the complexity around rather than eliminating it. 

    Introducing Turbulence

    Here is how Netflix, the organization that pioneered chaos engineering, defines the practice: “Chaos engineering is the discipline of experimenting on a distributed system to build confidence in the system’s ability to withstand turbulent conditions.” Turbulence is the opposite of order, but it’s where all systems end up.

    In the beginning of a system’s life cycle, there is always an orderly plan for deployment and operations. You always have a clean diagram of your resources, code repositories, containers and always a nice 3D diagram of AWS instances you will deploy to and the network topology. 

    In reality, our systems are never this simple. After the system goes live, that is when we start learning about how it actually operates through unforeseen events like when marketing asks to refactor the pricing module or an outage to the payments API forces you to hardcode a token or a DNS outage—well, you know about DNS outages. In the span of a year or two, you go from 300 to 800 microservices and you have to add Kubernetes clusters and a container network interface (CNI) to properly secure the networking of those clusters. And so on and so on. The bottom line? Almost all production systems evolve into a state that we do not understand and cannot easily map back to our original systems diagram. 

    Chaos engineering is a mechanism for dealing with this growing level of complexity. The basic idea behind chaos engineering is asking the computer or application or network (or any other environmental element) a series of questions. Consider a legacy system like a mainframe. How did our legacy systems become so stable? They are well-documented. Engineers feel confident in how the system operates. The question is, was the system always that way? Is the system the same one that they originally designed? The answer is “No.” 

    Legacy systems actually went through a trial-by-fire; a constant parade of unforeseen surprises. These surprises were, effectively, questions. Through these surprises, engineers learned how the system would actually operate (or wouldn’t operate) in the face of chaotic and unforeseen events. This process created iterative feedback that helped the engineers learn how to make the system continuously more stable. 

    The point of chaos engineering is to generate this same type of feedback loop without the pain, stress and repercussions of unforeseen (and generally unwanted) events. Thus, chaos engineering is a proactive way of introducing adverse conditions to ensure that the system can still do what it’s supposed to do when unforeseen events transpire. Because everything today is software and change is constant, the questioning must also be constant. 

    For example, chances are that in the time since we originally designed any given system, it has changed a lot. This means some or all the mitigation and resilience we programmed in—retry logic, failovers, circuit breakers—are old news, designed for an older system. When triggers for those conditions actually occur, those mechanisms of resilience we designed may not perform as we had intended.  

    Chaos engineers proactively ask the question, “Under the conditions in which you are designed to perform, do you do what you are supposed to?” We introduce the conditions themselves in a controlled manner with a limited blast radius to put the system to the test and verify that our assumptions line up with the intended reality. The practice of chaos engineering is intended to be performed on live, running systems without the risk of taking down critical production assets or ambushing unsuspecting applications and their associated DevOps and development teams. This helps engineering teams reduce uncertainty by building confidence in their understanding of their systems by providing them with enhanced context. It turns out that, when engineers are presented with better context, it considerably improves their ability to make smarter, well-informed decisions where resilience is concerned. 

    Chaos Engineering for Cybersecurity

    Cybersecurity is a context-dependent discipline, dependent on how well you understand the systems you are protecting.  Software engineers are constantly changing the system. And they must have the flexibility to change it because their primary task is to constantly move towards business value. Yet to effectively apply cybersecurity practices, we must know what we are trying to secure. From that knowledge, we then can know what we most need to secure and how to do it on a given system in its most current state. 

    Before an engineer can understand how to build robust security measures that provide effective cybersecurity, the engineer must first understand the system. This reliance on the “system first” is a foundational driver for why our security systems and controls—even the immutable and ephemeral ones—experience considerable engineering drift over time. We are continuously burdened by the dependence on a stateful understanding of the system.  

    Without having some sort of testing or instrumentation feedback loop in our post-deployment world, we don’t know there is a problem with our security until there is an actual problem. Finding out that the room is on fire after the house has been burning for hours is too late. The result is the self-defeating series of reactive incident fire drills, war rooms and outages that we find ourselves in today. If we do not change this reactive crisis loop of chasing the tiger’s tail, it inevitably leaves us taking two steps back for every step we take forward with engineering teams. 

    When this drift goes undetected, we are left with our wounds exposed and attackers salivating at the meal to come.  Engineers always prefer to find evidence and sources of drift through some internal engineering failure or controlled test rather than experiencing a successful real-world attack. Over time, there is a constant drift between the system and security; engineers are constantly changing the system. As a result, there is frequently a lack of recalibration between systems and the security designed to protect them. The way to achieve this recalibration is through continuous testing or experimentation. 

    Testing is not chaos engineering. Testing is a verification and validation of something you know as either true or false. This could be vulnerability to a specific CVE or exposure to an attack pattern. In experimentation, you are trying to derive new information and explore the unknown unknowns. From this exploration, you seek to create a  better understanding of how a system reacts and then share that with the people who are designing the system.

    Live security incidents are not effective measures of detection because, by then, it is already too late in the life cycle to effectively address the problem. Worse, incidents cause stress, fear and irrational behaviors. People freak out. Not surprisingly, a security incident is not a good learning environment. Yet security incidents serve the same purpose: Asking questions about a system or systems and helping teams see how to improve them. Chaos engineering offers a better way to create these questions under conditions where it is easier to learn and observe system behaviors, providing clues to improving security. In other words, the purpose of security chaos engineering is to help us proactively understand our systems and where the security gaps are before an adversary has a chance to take advantage of them.

    There are multiple use cases for security chaos engineering including:

    • Incident response: Testing efficacy of run books, response plans, and determining whether data collection and alerting systems work as intended
    • Security control validation: Introducing conditions that should induce behavior from one or more controls and observing behavior
    • Security observability: Introducing unexpected attacks or breaches to see if your systems report and observe the incident accurately
    • Compliance monitoring: Applicable across all chaos engineering events by documenting that systems do what they say they are supposed to do

    A Practical Example

    A team I headed wrote an open source application, ChaoSlingr, designed to generate chaos engineering events for security experiments. The application consisted of three serverless functions to generate the experiment, apply it to infrastructure and then monitor the resulting impact on the system. 

    One example of how we used ChaoSlingr was an exercise where we induced an open port. Enterprises and engineers inadvertently open ports in firewalls all the time. Sometimes this is a result of misconfiguration. It may also be the result of a software upgrade after which the developer or AppSec engineer forgot to reset the port to the proper security settings. In theory, firewalls should automatically detect an unauthorized open port.

    For the experiment, we selected a set of resources that had opted-in to our experiments, selected a security group and opened the port. At the time we were relatively new to Amazon Web Services. We were surprised to find that the firewall only detected an unauthorized port opening 60% of the time. We discovered clear evidence of configuration and security tool drift. 

    We also noticed that a cloud-native configuration and management tool we had deployed was identifying and alerting the unauthorized port opening almost 100% of the time. We then realized that correlating and responding to incident events of an unauthorized port opening was complicated; it was hard to map the incident back to the log data describing how our systems responded to the event. We added a metadata pointer to each alert, allowing us to more easily correlate the information and save valuable time finding the affected security group and port. 

    In short, the exercise gave us excellent context on where our systems were drifting and failing to keep up, which of our systems were the most reliable and whether we could quickly and easily respond to these incidents by chasing down the right log files. 

    Security Chaos Engineering Programs Build Trust

    In any environment where we are managing complex systems, trust is critical. You must trust that your systems will handle unforeseen events gracefully and not cost your company money or reputational currency. You must trust that the people designing and building your systems are constantly trying to improve them.  You must trust that the security teams protecting your systems are up to the task. As systems grow more and more complex, the number of unknown unknowns multiplies. As humans, with our limited processing capacity, we cannot solve for unknown unknowns at scale, continuously. This is the task of security chaos engineering and the processes around it: To create a programmatic way to experiment against and find answers to unknown unknowns. Your systems, developers, customers and your CEO will thank you for it.

    About Secure Software Summit
    The Secure Software Summit brought together the world’s leading innovators, practitioners and academics of secure software development to share and teach the latest methods and breakthroughs on secure coding and deployment practices. If you are about developing, releasing and securing software, delivering new features fast and building things right from the start, click on the link below to get access to all of the sessions from this summit.

    https://go.shiftleft.io/secure-software-summit-2022-replay

     

  • A Closer Look at Azure DevOps

    A Closer Look at Azure DevOps

    Microsoft’s Azure DevOps Server is emerging as one of the most popular cloud application development environments. Like most continuous integration/continuous delivery (CI/CD) platforms, it provides features such as version control, reporting, project management, automated builds, and testing. It covers all aspects of the application life cycle and enables DevOps. Here’s a closer look at this cloud computing environment.

    Azure DevOps provides a number of different developer services that allow DevOps teams to plan work, collaborate on code development and build and deploy applications. It is highly collaborative and brings together developers, project managers and individual contributors to develop software. It allows organizations to create and improve products much faster than with traditional software development approaches.

    Services or Server?

    Teams can work in the cloud using Azure DevOps Services or on-premises using Azure DevOps Server.

    The cloud offering, Azure DevOps Services, is a scalable, reliable and globally available hosted service. It’s backed by a 99.9% SLA, is monitored by Microsoft’s 24/7 operations team and is available in local data centers around the world.

    The on-premises offering, Azure DevOps Server, is built on a SQL Server back end. The on-premises version is preferable when organizations need their data to stay within their network or when they want access to SQL Server reporting services that integrate with Azure DevOps Server data and tools.

    Although both offerings provide the same essential services, Azure DevOps Services offers some added benefits:

    • Simplified server management.
    • Immediate access to the new features and updates.
    • Improved connectivity with remote sites.
    • A transition from capital expenditures (servers, etc.) to operational expenditures (subscriptions).

    There are also some other key differences depending on your organization’s or your team’s needs. These can also be helpful if you’re migrating from on-premises to the cloud:

    Differences in Specific Feature Areas

    Although Azure DevOps Services is a hosted version of Azure DevOps Server, there are some feature differences. Some Server features aren’t supported in Services; for example, Azure DevOps Services doesn’t integrate with SQL Server Analysis Services to support reporting. Process customization and reporting also differ in their support.

    Integrated Features

    Azure DevOps provides integrated features that can be accessed through a web browser or an IDE client. One or more of the following standalone services can be used based on your organization’s business needs:

    • Azure Repos: Azure repos provides Git repositories or Team Foundation Version Control (TFVC) for source control.
    • Azure Pipelines: This service provides build and release services to support continuous integration and continuous delivery of applications.
    • Azure Boards: This suite of Agile tools supports planning and tracking work, code defects and issues using Kanban and Scrum methods.
    • Azure Test Plans: Azure Test Plans provides several tools to test applications, including manual/exploratory testing and continuous testing.
    • Azure Artifacts: These allow teams to share packages such as Maven, npm, NuGet and more from both public and private sources and integrate package sharing into pipelines.

    Use of the following collaboration tools is also supported:

    • Customizable team dashboards with configurable widgets to share information, progress and trends.
    • Built-in wikis for sharing information.
    • Configurable notifications.

    Azure DevOps supports adding extensions and integrating with other popular services, such as Campfire, Slack, Trello, UserVoice and more. In addition, DevOps teams can also develop their own custom extensions.

    Azure DevOps Services supports integration with GitHub and GitHub Enterprise Server code repositories. Azure DevOps Server supports integration with GitHub Enterprise Server repositories.

    Getting Started

    When you sign up for Azure DevOps, you must first register an organization, project and version control type. DevOps teams can create multiple projects and differentiate them with separate names and access permissions. Projects can be set to public or private. Teams can also use Azure Boards for collaboration and project management, and can also configure use of Agile practices. After setting up your team, you can use the platform to begin deploying your applications to Azure.

    In addition to a collaborative environment, Azure DevOps provides access to an Azure Pipeline service that supports multiple languages and platforms. The Azure DevOps pipeline is a set of services that are grouped into individual projects. 

    Additionally, Azure Test Plans feature facilitates automated testing. It is also possible to set up deployment pipelines with different components for multiple applications. 

    Azure Pipeline can track work status, manage backlogs and create custom reports. With the help of the Azure DevOps pipeline, developers can create, manage and deploy their projects in a single environment. With this solution, DevOps teams will have a single point of contact and can communicate with each other.

  • Secure Software Summit: Measuring and Mitigating OSS Risks

    Secure Software Summit: Measuring and Mitigating OSS Risks

    Measuring and mitigating the security risks in open source software is becoming a major issue in the software development community. Attacks on open source software (OSS) are on the rise; open source supply chain attacks grew 650% in 2021—a staggering number.

    2021 and early 2022 saw significant attacks which impacted more than 1,000 organizations. The most prominent recent OSS attack was the Log4Shell vulnerability disclosed in December 2021. Attacks on the OSS supply chain and on OSS impacted nearly every single software organization on the planet. Everybody is scrambling to continuously patch systems as new vulnerabilities continue pouring in.

    To properly navigate this environment and keep our systems safe, we as a community need to understand the security risks of OSS and be better prepared to handle them, collectively and individually. 

    Open Source Risks

    There are a number of risks that make OSS more vulnerable without proper security measures. 

    OSS is Free; OSS Security is Expensive

    To start, we must recognize that open source software may be free to download and require no payment, but it comes with a massive security cost. 

    A significant portion of open source software is developed by volunteers in their free time. Most of the prominent open source volunteers actually have full-time jobs, and the code they contribute to projects like the Log4j logging libraries are written on nights and weekends; on their own time and not on the company dime. Not surprisingly, security isn’t high on their priority list. They don’t have the resources to invest in a secure development life cycle or have the time and bandwidth to maintain their code. This may mean that security vulnerabilities remain unaddressed for long periods of time, sometimes indefinitely. 

    All Bugs are Shallow, but There Aren’t Enough Eyeballs

    In software development, Linus’s law states “…given enough eyeballs, all bugs are shallow.” However, this is only true if those eyes are looking in the right place. In the case of open source, all code is open but there aren’t enough eyes paying attention to it to spot all the bugs. In addition, organizations may not consider the security consequences of their decision to consume a particular piece of open source software and adding it as a dependency. Dependency trees become more and more complex; harder and harder to monitor and secure, as you keep adding more third-party components to a core codebase. Each new dependency might bring additional nested dependencies—perhaps hundreds or thousands, depending on the size of the library added or the role it plays in the application architecture. For example, the Kubernetes project—one of the most successful and widely adopted OSS projects—has several thousand dependencies, many of which have additional secondary and tertiary downstream dependencies. 

    So Much Code. So Many Choices. Not all are Good.

    The beauty of the open source ecosystem is that there is a solution for everything. The problem with the open source ecosystem is that there’s not just one solution, but several dozen. For example, if you need a JSON parser for your application, there are several to choose from. If your application is running a critical service or infrastructure, then picking the right JSON parser—one that is secure and comes from a responsible project or maintainer—becomes very important. A JSON parser will serve as a critical component for your application, so it will impact the security of your entire system. If you chose the wrong JSON parser, you might pay a big price down the road in security risk and subsequent breaches. 

    Proposal: An Entry Gate Security Risk Assessment

    One potential solution to enhance OSS code security is the creation and implementation of a security risk assessment gate as part of your OSS ingestion pipeline. Putting this process in place will help you understand whether consuming a certain piece of software is high risk and avoid software that does not meet your risk criteria. 

    Here is an example of how this might work. Consider five different JSON parser libraries in GitHub, all of which have a high number of stars and appear to be highly rated. How can we assess the risk of these libraries and quickly and easily eliminate ones that are too risky? One of these JSON libraries is hosted under a user account. Most of the commits are done by a single developer without reviews. This alone indicates that using this library in a critical system presents a high risk. This lone developer could commit malicious code without review. The maintainer’s token access could be leaked, allowing attackers to insert malicious code. The developer might also turn over committer control of their library to a malicious actor without notifying users (who might not pay attention to the notification anyway). Identifying this risk required no more than a few minutes of review of the repository and cost little in time or effort. 

    A Broader View of Security Posture for OSS

    More broadly, we need to understand the full security posture of an organization or developer and their open source code prior to adding it to critical systems. Because OSS is a complex ecosystem and many pieces of OSS are complex, the risks they generate are similarly complex. Our traditional, myopic view of these risks is that we just need to find all the security vulnerabilities in our codebase and fix them. This does not resolve all risks and may lull us into a false sense of security.

    We need to understand the full comprehensive security posture of an open source project. That involves understanding a broad range of risks beyond code vulnerabilities. Here are some of the key risks to consider beyond ensuring that your codebase is well-secured and patched. 

    Supply Chain Attacks on the Build Pipeline

    Attacks like SolarWinds and CodeCove taught us that supply chain risk is a very big deal and that tools are just as much at risk as code. Attackers have found it much easier and much more reliable to attack build pipelines, rather than finding vulnerabilities in code.

    Security Risks Revealed by Continuous Security Testing 

    Continuous security testing is critical to maintaining the security of a codebase. There might not be code vulnerabilities today, but as developers continuously modify code and iterate on applications, vulnerabilities may be introduced. This can mean testing code as part of the CI/CD process with every new code push. For example, we have found numerous critical bugs in OpenSSL libraries. All were caught on a master or main branch and never made it to a production release. 

    Project Maintenance and Capabilities

    One of the really big risks with open source is the quality and frequency of project maintenance. Issues raised may not be addressed in a timely fashion. Maintainers of a project may have a long backlog of issues, indicating that they are not actively maintaining a project. It is not uncommon for critical open source projects used in millions of applications and comprising a significant dependency to the global software supply chain to not have been updated in a year or more. In addition, maintainers of OSS projects may not know how to respond to security incidents and fix security vulnerabilities. This is particularly true if the maintainers do not work at a large organization, do not have cybersecurity or secure development experience or have never suffered through the toil and trauma of addressing a major vulnerability in their code on short notice. 

    Tools for Measuring and Addressing Supply Chain Risk

    A number of efforts are underway to create free tools to help developers and application security teams better measure and address supply chain risk programmatically and analytically. 

    SLSA—Supply Chain Level for Software Artifacts

    As a tool to help organizations better address software supply chain risks, Google created a novel framework that we call Supply Chain Level for Software Artifacts (SLSA). The project has been underway at Google for more than a decade and has been open sourced as part of the OSSF. Contributors include VMware, Citibank and Datadog, among others. Google hopes to standardize SLSA and make it an industry benchmark for evaluating supply chain risk. Right now, SLSA is focused primarily on build process risks but the hope is to expand it in the near future to other areas, such as vulnerability and maintainer risks.

    OSS Security Scorecards Project

    In order to scale the process of assessing, identifying and mitigating risks in OSS, we need robust automation. In 2020, as part of the OSSF, Google released a new project called Open Source Security Scorecard. The project aimed to automate the analysis and trust aspect when evaluating the security posture of open source code. Further, the Scorecards can provide specific security recommendations. The Scorecard process can incorporate many other automated tools, including checking against vulnerabilities using the Open Source Vulnerability (OSV) database and running dependency checks using open source dependency trackers like Dependabot. The scorecards project also makes it easier to report new vulnerabilities and get those to the right people in a project. 

    Conclusion: What is not Measured is not Mitigated

    No one denies that OSS has become crucial to the vast majority of applications running the world’s technology foundation. The span of OSS continues to grow, as more and more languages and frameworks emerge and specialization increases. No one wants to reinvent the wheel and rewrite fresh code to perform the same tasks with every new application. The model of reuse, dependency and distributed software development works and speeds innovation. That said, the open source community and the supply chain used to build OSS need a more rigorous approach to reducing risk. Given the scope of the problem, automation and standardization are helping to scale toward the stated goal. We are in the early stages of creating the tools and infrastructure to automate the risk identification, measurement, assessment and mitigation processes. Security frameworks like SLSA and new processes like entry gating new OSS code with security scorecards will direct organizations down that path and move us toward a more secure open source future.

    About Secure Software Summit
    The Secure Software Summit brought together the world’s leading innovators, practitioners and academics of secure software development to share and teach the latest methods and breakthroughs on secure coding and deployment practices. If you are about developing, releasing and securing software, delivering new features fast and building things right from the start, click on the link below to get access to all of the sessions from this summit.

    https://go.shiftleft.io/secure-software-summit-2022-replay

     

  • CircleCI Report: Pipeline Optimization Best Practices

    CircleCI Report: Pipeline Optimization Best Practices

    CircleCI today published a report that found most successful software engineering teams have average workflow durations of five to 10 minutes, with any failed run being fixed or reverted in under an hour.

    The report was based on two years of data gathered from more than a quarter-billion workflows and collected from nearly 50,000 organizations using its continuous integration/continuous delivery (CI/CD) platform. The CircleCI 2022 State of Software Delivery Report also found those teams have success rates that are above 90% for the default branch of their application and deploy at least more than once per day.

    Mike Stahnke, vice president of platform for CircleCI, said the report also makes it clear there is no correlation between team size and the level of DevOps success achieved.

    Organizations that focus on optimizing pipelines are also better able to integrate testing within their DevOps workflows, he noted.

    In general, Stahnke said more DevOps teams are relying on cloud-based CI/CD platforms versus opting to deploy and maintain infrastructure themselves. That shift also lowers the barrier to entry for organizations looking to embrace DevOps best practices to accelerate application development, he added.

    Less clear is the degree to which organizations have embraced both CD and CI. While many developers routinely employ CI to accelerate application development, CD has proven to be a more elusive goal. Each platform a DevOps team supports tends to have unique attributes that require them to rely on manual processes to deploy applications. However, as DevOps processes continue to mature, Stahnke said the CI and CD integration will become more seamless. In fact, as the number of applications that organizations build simultaneously increases, the pressure to automate CD will also rise.

    However, it remains to be seen to what percentage of organizations will opt for a tightly integrated CI/CD platform versus separate CI platforms for developers and CD platforms for IT operations teams that are more loosely coupled.

    In the meantime, organizations of all sizes are clearly more focused than ever on developer productivity. In the wake of the downturn brought on by the COVID-19 pandemic, there was a rush to launch a range of digital business transformation initiatives. The issue that many organizations are now encountering is they need to find ways to enable a finite number of developers to simultaneously build and deploy multiple applications with multiple interdependencies. Many organizations are trying to address that challenge by sharing best practices across DevOps teams that have varying levels of maturity. Few organizations have a set of standardized DevOps practices that are consistently employed across every application development team within an enterprise.

    It will be up to each organization to decide whether to compare their level of DevOps expertise to other organizations. Each application development team faces a unique set of challenges. The most important goal is simply to continuously improve at a time when demand for custom applications has never been greater.

  • What Does AIOps Mean for SREs?

    What Does AIOps Mean for SREs?

    It seems SREs are of two minds when it comes to AIOps. On one hand, AIOps’ potential is pretty exciting. By automating complex workflows and troubleshooting processes, AIOps could make your life as an SRE much easier.

    But on the other hand, some SREs may choose to view AIOps with disdain and distrust. They might think of AIOps as just another trendy buzzword that doesn’t live up to the hype and that can become a distraction from the SRE tools that really matter.

    Which perspective is right? Should SREs embrace AIOps with open arms, or should they resist marketers’ efforts to position AIOps as the latest, greatest tooling innovation in the IT industry?

    Those are subjective questions that we can’t answer definitively, but let’s at least gain some perspective by examining what AIOps means for SREs.

    What is AIOps?

    If you keep up with IT buzzwords, you’ve probably at least heard the term AIOps—which is short for artificial intelligence for IT operations—and that it refers to the use of AI and machine learning (ML) to help automate IT operations workflows.

    The big idea behind AIOps is that, by using AI and ML to perform advanced analysis of large volumes of data from IT systems, IT and SRE teams can solve complex problems more efficiently than they could using a traditional, manual approach.

    AIOps can, for example, help to surface the root cause of a performance issue in a complex, multilayered environment like Kubernetes. Or it could make recommendations about how best to resolve an incident.

    AIOps entered the IT lexicon in 2016, when Gartner coined the term. Now, six years later, it’s a relatively well-established tool domain.

    How SREs View AIOps

    Despite the fact that AIOps has been around for some time, it doesn’t yet appear that many SREs have bought into the AIOps revolution. Catchpoint found in a 2021 survey that just 7.5% of SREs reported that AIOps tools delivered high value to their organizations.

    It’s unclear exactly why there are such low rates of excitement about AIOps, but we’d speculate that there are a few key factors at play:

    • AIOps is a new term for an old idea: Many monitoring and observability tools have included at least basic AI and ML analytics features for a long time, even before tool vendors slapped the AIOps label on their products. SREs probably realize this and, to some extent, view AIOps as an effort by marketers to rebrand functionality that is not actually new.
    • AIOps is hard to implement: Setting up an AIOps tool requires integrating it with diverse data sources and customizing it to fit your workflows and environment. It’s possible some SREs view this setup work as requiring more effort than it’s worth.
    • AIOps can’t replace human insight: While AIOps tools may be useful to a point, it would be unwise to place blind trust in AIOps-based analyses or recommendations. For this reason, some SREs may believe that AIOps encourages organizations to rely too heavily on automated tools at the cost of the expert analysis and perspective that only SREs can provide. (This is similar to the fact that it sometimes makes sense to prioritize human intuition and expertise over playbooks.)

    From an SRE’s perspective, then, AIOps may appear over-hyped, overly complicated and underperforming compared to traditional approaches to SRE.

    What SREs can Gain From AIOps

    SREs’ wariness toward AIOps is valid—but only to a point. It’s important not to let suspicions about the limitations of AIOps turn into excuses not to use AIOps at all. AIOps has some value to offer to SREs, even if it’s not perfect.

    For example, AIOps can play a role in reducing toil. To the extent that AIOps tools can recognize complex patterns or discover interrelated data sets more quickly than human engineers, AIOps reduces the time SREs have to spend manually troubleshooting problems or poring over complicated information.

    AIOps also helps to enable a more proactive approach to monitoring and incident management. If AIOps tools can alert SREs to emerging issues before they would otherwise recognize them, AIOps can help the SREs to get in front of the problems before they turn into true incidents. That’s better for SREs and end users alike.

    There is also an argument to be made that AIOps can help SREs do more with fewer engineering resources. If you can use AI to automate some aspects of monitoring and incident response, you can maintain the same levels of availability and performance with fewer human engineers on hand.

    Conclusion: AI Won’t Replace SREs—But it can Help

    None of the above is to say that AIOps can replace SREs, or that it magically solves every problem SREs face. Anyone who believes AIOps is a silver bullet has bought into the marketing hype to an unhealthy degree.

    Nonetheless, AIOps tools do offer value to SREs by making their jobs easier in some respects and improving reliability outcomes.

    So, while it’s wise to maintain a healthy perspective about the limitations of AIOps, SREs shouldn’t rule out AIOps tools as one way to improve reliability engineering.

  • CircleCI Acquires Ponicode to Gain AI Testing Tool

    CircleCI Acquires Ponicode to Gain AI Testing Tool

    CircleCI today revealed it has acquired Ponicode to gain control over a unit testing platform infused with artificial intelligence (AI) that makes it simpler to review application code prior to uploading it into a build. Financial terms of the deal were not disclosed.

    Jean-Philippe (JP) Leblanc, senior vice president of engineering at CircleCI, said the goal is to integrate Ponicode’s testing technology within CircleCI’s continuous integration/continuous delivery (CI/CD) platform. That approach will make it possible to apply algorithms to individual developers’ code as well as to code aggregated from multiple developers within a build as it is continuously updated across a DevOps workflow, he added.

    Ponicode was initially launched in June 2020 as a JavaScript unit testing extension for Microsoft VS Code. Since then, the company has started to support TypeScript, Java and Python, including complex functions such as variable mocking and infer suggestions of complex structures for nested objects.

    The issue that DevOps teams frequently encounter is that even when individual developers test their code prior to uploading into a build, integrating different developers’ code can still result in an application failure. The Ponicode AI testing tools will give both developers and IT operations teams more confidence in the quality of the code being employed to drive applications, noted Leblanc.

    Although not many developers want to admit it, Leblanc noted there is still a lot of sloppy code finding its way into applications.

    In addition, Leblanc said the Ponicode acquisition will help the code review process become more efficient. Instead of requiring another developer or dedicated tester to review code, he said most issues will be surfaced by the AI testing tool that provides the equivalent of a built-in peer review. As a result, AI testing tools will improve the overall quality of the code being used to drive applications, added Leblanc. Developers will also be able to spend more time on the business logic that drives their application than on code reviews.

    The goal, however, is not to eliminate the need for dedicated testing teams as much as it is to augment them and the developers they work with, said Leblanc.

    It’s not clear to what degree responsibility for testing code is shifting left toward developers. However, many developers would prefer to have issues surfaced via an AI tool so they could address issues on their own rather than having them discovered by someone else. In fact, there are already more than 20,000 users of the Ponicode AI tool that plugs directly into developers’ integrated development environment (IDE), said Leblanc.

    In general, there’s never been a greater focus on developer productivity. Organizations are attempting to simultaneously build and deploy more applications than ever—thanks mainly to digital business transformation initiatives launched in the wake of the COVID-19 pandemic. The challenge they face is recruiting and retaining developers. As a result, organizations are looking for ways to increase developer efficiency by eliminating as many manual processes as possible.

  • Why Software Usability Testing Is Essential

    Why Software Usability Testing Is Essential

    Keeping an eye on the bigger picture is important when doing just about anything. This adage holds true when testing software, too. There’s no doubt that the QA testing process for software is essential. It makes sure everything is running smoothly and as it should. But generally speaking, it doesn’t always fully take into account the user perspective. Even though developers, designers and testers all try to keep the end user in mind when building and testing, there is no true way to anticipate and test for the variability of actual user experience in the wild.

    This is why it is essential to consult actual users. That’s where the process of usability testing comes in. Every organization should consider implementing the practice before a software product launch.

    Usability Testing Defined

    Unlike functional testing, usability testing hones in on the user’s experience of a product. It focuses on how an application works in the real world with real users. In-house, it’s easy to test a feature and check a box to say it works. But that’s not necessarily the way an application will be used by every user in every situation.

    Usability testing involves having participants complete a few tasks with an application. Then, depending on how easy or difficult those tasks were, testers and developers can decide which, if any, improvements and changes need to be made. Usability testing helps organizations accomplish three key goals:

    1. Uncover opportunities to improve
    2. Identify issues in design
    3. Learn about intended user behaviors

    Differences Between Testing Types

    Software usability testing is sometimes confused with other similar approaches to quality assurance and control. Let’s look at and clarify similarities and differences between these methods.

    • User research versus usability testing: User research comes earlier in the software development life cycle and actually involves usability testing, but it also includes other components like interviews, surveys and focus groups to identify the behaviors of users.
    • User testing versus usability testing: User testing is put into place to validate a product idea. It’s used to understand how a user approaches a given situation and how a product might help address that situation. Usability comes later in the process of testing that specific product.
    • User acceptance testing versus usability testing: User acceptance testing confirms that a product works as intended; it’s about functionality. But usability testing will get into the specifics of user opinions, expectations and actual experience.

    Types of Usability Testing

    There isn’t just one way to conduct usability testing. There are a few options organizations can choose from based on differing objectives.

    • Moderated versus unmoderated: Moderated usability testing involves a moderator facilitating the progress of a tester through a series of tasks. This gives an organization the opportunity to ask questions during specific parts of the process. An unmoderated approach allows a user to complete tasks entirely at their own pace. This method can be conducted at any time, anywhere.
    • Remote versus in-person: In-person testing gives UX researchers the ability to provide specific devices to avoid technical issues, although it might mean missing issues with a user’s actual device that occur in the wild. It also allows for observing non-verbal reactions. Remote testing, on the other hand, allows users to complete tasks as they might in the real world from anywhere and with their actual device.

    Some specific usability testing methods and techniques that can be used include:

    • Screen/video recording: Gives insight into how long it takes to navigate and click through a website, application, system, etc. as well as tracking non-verbal cues.
    • Usability lab: A dedicated testing location. All variables can be controlled so the focus remains on the testing.
    • Guerilla testing: This is done in the wild, recruiting a few people to do a quick software test. While this can be quick and cheap, you can’t ensure you’re testing your specific target audience.
    • Contextual inquiry: Testing in a participant’s own environment with an interview and/or third-party observation of their use of the product. This allows researchers to know things like what device is used and where and how the application is used.
    • Usability testing partners: These are third-party organizations with testing expertise and access to a broad pool of participants that can be recruited and cultivated based on client needs.

    Benefits

    Usability testing can deliver a variety of benefits to an organization, including:

    • Direct feedback from the target audience
    • Cost savings through addressing concerns earlier in software development
    • Detection of usability issues before launch
    • Real-world testing
    • Unbiased assessment

    Costs and Challenges

    The most obvious concerns at the beginning of the software development life cycle are are development costs and time-to-market. While usability testing can take time and involve costs of its own, ultimately, the practice aims to address those concerns; in the long run, it is more expensive to launch a flawed product with poor user experience. A few factors that must be considered when it comes to the time and cost involved in performing usability testing include:

    • Sample size
    • Gathering the right participant profiles
    • Breadth and depth of testing practices
    • Amount of time allocated to the test

    All things considered, when done right usability testing delivers ROI that more than makes up for the cost and time involved.

    Consider the implications when an application is launched and certain users are frustrated with the experience and are left with a negative feeling toward the brand. Usability testing addresses this concern and allows for issues to be discovered and fixed earlier in the development life cycle. Investing in developing and testing for a positive user experience is always worthwhile and effective.

  • Tabnine, GitLab Build Custom Apps Faster With AI

    Tabnine, GitLab Build Custom Apps Faster With AI

    Tabnine today revealed it has integrated its code completion tool with the GitLab continuous integration/continuous delivery (CI/CD) platform. Tabnine’s platform employs artificial intelligence (AI) to accelerate application development.

    Brandon Jung, vice president of ecosystems and business development at Tabnine, said the goal is to make it easier for developers to automatically write code based on custom models created using AI. Those models would reside in the CI/CD platform provided by GitLab. Developers can now create custom models of their private source code, he noted.

    The Tabnine AI assistant can then be trained on a customer’s private code, style and security best practices that have been defined by an internal DevOps team, added Jung. That assistant, accessible from within a developer’s integrated development environment (IDE), is conversant in 35 programming languages.

    The code completion tool doesn’t eliminate the need for developers to write code; instead, it completes code once developers start writing it based on the model stored in the CI/CD platform.

    Nima Badiey, vice president of global alliances for GitLab, said the overall goal is to increase developer productivity at a time when organizations are launching more application development initiatives than ever. Many of those applications are being constructed using multiple programming languages that each individual developer may not have equally mastered.

    In addition, an AI model will, over time, improve the quality of the code being created, noted Badiey. That should make it easier to review code that is more accurate because it was created with the aid of a machine, he added.

    In general, GitLab has been moving to incorporate AI within DevOps workflows. The company last year acquired UnReview, a provider of a tool that uses machine learning algorithms to identify which code reviewers to assign to a project based on both the quality of their previous efforts and their current workloads.

    It’s not clear to what degree AI will be pervasively embedded within DevOps workflows. However, it’s now just a matter of time before algorithms are more widely employed. Organizations that employ DevOps today devote significant resources to maintaining platforms that could be simpler to manage if more tasks and processes were automated. In theory, more resources could be then devoted to building applications versus maintaining DevOps platforms.

    One way or another, more organizations are moving to increase developer productivity. Developers are both hard to find and retain, so organizations have a vested interest in finding ways to increase the rate at which an existing pool of developers can build applications capable of driving a wider range of digital business transformation initiatives. Of course, an increase in the rate at which applications are developed also has significant implications for the rate at which they might need to be deployed.

    Not everyone in IT is, of course, entirely comfortable with AI. Nevertheless, most DevOps organizations are highly committed to automating as many processes as possible. The only thing that will be different going forward is determining to what degree those processes will be automated by humans versus a machine.