Category: Continuous Testing

Continuous testing is the process of executing automated tests as part of the software delivery pipeline to obtain immediate feedback on the business risks associated with a software release candidate.

  • The Scariest Things About SCA

    The Scariest Things About SCA

    It is a time of ghouls, mischievous spirits and David S. Pumpkins. In the spirit of Halloween, here are the top five scariest limitations of software composition analysis (SCA) tools that are enough to send shivers down your spine. Read on … if you dare!

    1. SCA Scans Only Your Application Code

    SCA’s scope is frightfully narrow. When scanning for vulnerable dependencies, SCA only examines your source code even though vulnerable dependencies used to build your software can exist throughout your entire development pipeline. Vulnerable dependencies can be found in dev tools like Jenkins, dev tool plugins like Jenkins Plugins, build modules like GitHub Actions, build module dependencies like GitHub Actions libraries, and infrastructure-as-code (IaC) dependencies. If you’re only scanning application code, you should be shaking in your boots because it is not enough to protect against modern software supply chain attacks like SolarWinds or Codecov.

    The good news is that next-gen SCA, pipeline composition analysis (PCA), scans your application code and your entire pipeline for vulnerable dependencies. PCA improves upon the foundation of SCA by including a deep understanding of the tools, configurations, processes, activities, components and dependencies used at every stage of the delivery pipeline. When new dependencies like Text4Shell are disclosed, you can be sure every instance of the vulnerability is found and remediated. This way you know you don’t have any proverbial skeletons lurking in your software’s closet.

    2. SCA Can’t Tell You Where Vulnerabilities Are Deployed

    Vulnerability remediation can be ghoulish. When new high-severity exploits are disclosed, you need to remediate every instance of the vulnerability from all production environments where it could be exploited–and you need to do it fast. Identifying a vulnerability like Log4j in your code base is a start, but you’re not truly secure until you have removed every instance of the vulnerability in production systems where they pose a real risk to your organization. Though traditional SCA can tell you where a vulnerability is in your application code, it has no visibility once that code has been deployed.

    PCA gives you visibility into your entire development pipeline, including production environments. This enables you to easily trace the path of a vulnerability from code to deployment locations, such as identifying which Kubernetes clusters contain a specific vulnerability. This enables organizations to be more agile in their response to newly disclosed threats and ensures that every instance of a vulnerability has been remediated.

    3. SCA Doesn’t Correlate Risk Across AppSec Engines

    Some traditional SCA solutions may be able to collect data from various scanning engines, but they are terrible at making meaningful connections between various data points. Because of this, “traditional” SCA is unable to provide the context required to deliver an accurate and nuanced assessment of the impact of one event and how risk might be compounded by other related events. SCA without context generates too many false positives—this results in your team wasting time chasing ghosts.

    PCA correlates risks and threats across all phases of the SDLC as well as from all AppSec scanning tools. By connecting the dots from numerous data sources, PCA helps your security and DevSecOps teams understand holistically how risks relate to each other so they can best tackle the most important security initiatives first.

    4. SCA Ignores Pipeline Security and Governance

    Modern development pipelines are complex and deeply interconnected–almost as if they are stitched and bolted together like Frankenstein‘s creation. Though managing security policies across the entire SDLC is an arduous and painful task, it is essential if you want to prevent software supply chain attacks. Unfortunately, traditional SCA solutions don’t even come close to helping you manage pipeline security and governance policies like enforcing least privilege or hardening authentication.

    Pipeline composition analysis identifies pipeline tooling misconfigurations and helps you implement best practices for security controls across your entire pipeline. Even better, to prevent your tooling from being bypassed, PCA alerts you when pipeline security controls like SCA scanning are removed or disabled.

    5. SCA Provides No Visibility into Your SDLC

    Traditional SCA solutions focus only on application code. They provide no visibility across your SDLC. They were not designed to scan development and deployment environments, so they cannot secure the later parts of your SDLC. SolarWinds was infiltrated via its TeamCity build server, which traditional SCA tools did not scan. So how can you protect your software supply chain if you can’t see it? Don’t be haunted by SCA tools’ inability to scan the entire SDLC.

    Vulnerable dependencies that impact your application can be found across the SDLC. You need to understand all development and deployment tooling–activity, settings, configuration, users and more–to protect your application. Pipeline composition analysis was designed from the ground up to understand your SDLC and bring context to your SCA findings to improve accuracy. It can give you complete visibility into the big picture to cut down on all the noise that traditional solutions generate.

    Pipeline Composition Analysis: Put Your Nightmares to Rest

    While it’s fun to be scared at Halloween, the last thing you want is to be terrified by your AppSec program. When the limitations of traditional SCA solutions threaten to keep you up at night, just remember that AppSec innovation requires the ability to do pipeline composition analysis across the entire SDLC, hence the shift from SCA to PCA. PCA plugs all the holes mentioned here, which hopefully can put even the worst nightmares to rest.

  • Making App Dev More Efficient

    Making App Dev More Efficient

    Applications of all kinds are continually popping up across various industries to help improve consumer customer experiences, provide entertainment, make certain tedious or error-prone tasks easier for workers and to help businesses improve their operations overall. In fact, there are over 5.4 million applications available across the Apple App and Google Play stores as of this year.

    As more and more individuals and organizations turn to applications for various experiences and to drive enterprise efficiencies, we need to take stock of the process we deploy in application development.

    The Inefficiencies of Application Development

    Today, it is estimated that there are more than 26 million software developers worldwide. Each of these developers brings a unique combination of creativity and technology acumen to the organizations they work for and the applications they create. However, the current application development process in many enterprises is riddled with inefficiencies that often require these talented developers to focus on tedious, mundane, non-innovation-related tasks. 

    Technical Debt: Renowned programmer Ward Cunningham first introduced the concept of technical debt in 1992, stating, “Shipping first-time code is like going into debt. A little debt speeds development so long as it is paid back promptly with a rewrite. Objects make the cost of this transaction tolerable. The danger occurs when the debt is not repaid. Every minute spent on not-quite-right code counts as interest on that debt. Entire engineering organizations can be brought to a stand-still under the debt load of an unconsolidated implementation, object-oriented or otherwise.” What many didn’t realize at the time was that technical debt would grow to be a major issue in software and application development years later. Recently, a new Scandinavian study found that developers dedicate 23% of their time, on average, to addressing technical debt. While this statistic alone should be alarming to enterprises—given their development teams are spending a quarter of their time on an issue that could have been avoided had a different approach been taken—the study also found that developers often have to introduce new technical debt in the process. 

    Maintenance: As with all technological equipment, applications require routine maintenance to ensure they remain working for the end user, provide ongoing value and are secure. However, unlike a car or kitchen appliance, applications see far more frequent updates from operating systems (OSes), cloud or hardware providers that need to be addressed by development teams. According to research from Stripe, developers spend more than 17 hours per week on maintenance issues like debugging or refactoring. For instance, take a look at Apple. From the time iOS 15 was released in June 2021 as a beta, there were more than a dozen updates made to the OS until early September 2022. That means that every single time the updates were released, developers had to spend time ensuring their applications were working correctly, were bug-free or complied with new security measures. When you add this to the fact that the average enterprise app has 110 third-party integrations, that is an extreme amount of maintenance.  

    The Reinvention of the Wheel: Of all the applications available today, many of them consist of the same foundational features and functionality—login, chat, social sharing, payments and more. What’s astonishing about this is that developers are frequently recreating each of these common features and functionalities from scratch every time they develop a new application. 

    It is clear that these tasks are highly inefficient, and it begs the question, “What can be done to address these inefficiencies?” 

    Prioritizing Innovation 

    It is not enough to just introduce some sort of technology in the hopes that it makes application development more efficient. It also requires an overall mindset change by enterprises. Yes, it is still of the utmost importance to get applications to market quickly and keep up with the pace of competition; however, innovation doesn’t have to be sacrificed to do so. 

    Innovation must be the key driver for enterprises and their development teams. Providing unique value and experiences to end users will truly set enterprises and their applications apart from the competition. But, the reality is that nearly 80% of technology leaders from around the world believe basic application maintenance and support are the biggest barriers to innovation. 

    Enterprises must be able to balance innovation, time-to-market and development processes that are free of technical debt and outrageous maintenance support. 

    Making Application Development More Efficient

    One way in which enterprises can balance innovation, commercialization, and development is through an emerging approach to application development known as features-as-a-service (FaaS). 

    A FaaS approach to application development enables enterprises to implement an agile and accelerated engineering process—similar to that of low-code/no-code without the constraints often seen with those vendors and their functionality today—without sacrificing quality, code or time. More specifically, it combines pre-coded “commodity” features with continuous and automatic updates that empower development teams to spend their time and creative minds on innovation. In most cases, this approach can help enterprises bring applications to market within weeks instead of months and limits the headaches on the backend for development teams down the line. 

    As enterprises work through the fourth quarter of this year, I implore them to reconsider their development process at a fundamental level and bring innovation to the forefront before the beginning of 2023. Time and resources are of the essence, and something must be done now to address the inefficiencies of application development. 

  • Sigstore Code Signing Service Becomes Generally Available

    Sigstore Code Signing Service Becomes Generally Available

    A free digital signing service for software created by the Sigstore open source community has become generally available this week via the cloud.

    Announced at the SigstoreCon event that occurred during the KubeCon + CloudNativeCon North America conference, the cloud service makes it possible for developers to both cryptographically sign artifacts and verify that the components used to build applications are secure.

    The Sigstore community operates under the auspices of the Open Source Security Foundation (OpenSSF), an arm of the Linux Foundation. Sigstore project sponsors, including Google, Red Hat, GitHub and Chainguard, have committed to operating a service with a 99.5% uptime and round-the-clock pager support. More than 70 organizations, including Shopify, Autodesk, Trail of Bits and Rancher Government Solutions are actively involved in maintaining and scaling Sigstore.

    More the four million Sigstore signatures have been logged using Sigstore thanks mainly to Sigstore support that is already included by maintainers of open source Kubernetes and Python projects. The npm community also announced it is working to integrate Sigstore in all packages used to build JavaScript applications.

    Bob Callaway, co-founder of Sigstore and technical lead and manager for the Open Source Security Team at Google, said this service will make it much simpler for developers to add digital signatures to code in a way that can be more easily verified within a software bill of materials (SBOM).

    The goal now is to expand the number of open source projects that will commit to employing Sigstore to better secure software supply chains, he added.

    The security of open source software became a major concern in the wake of the open source Log4j zero-day vulnerability. Sigstore was under development prior to that disclosure, but the Biden administration’s executive order requiring Federal agencies to verify the security of open source software has enabled the availability of more resources for the project.

    In response to the executive order, the OpenSSF outlined a 10-point plan to secure open source software that would require more than $150 million in funding. One of those goals is to have 50 of the top 200 projects and 1,000 of the top 10,000 projects adopt an interoperable approach to software signing. That project was estimated to cost $13 million for the first year and $4 million per year beyond, with a one-time additional $10 million required after the first year.

    It’s not clear how broadly developers will employ digital signatures, but given the current level of concern, it’s probable most organizations will soon require any code they use to be digitally signed so it can be verified as immutable. The next big challenge, of course, is finding a way to operationalize the collection of all the SBOMs that will include those signatures. After all, SBOMs are only the first step in a larger DevSecOps workflow that will enable organizations to accept or reject software components based on all the data being collected.

  • Software Quality is the Heartbeat of the Best Organizations

    Software Quality is the Heartbeat of the Best Organizations

    JPMorgan recently announced it was hiring 2,000 engineers, despite the gloom in global economic markets. Is this not an odd risk for an organization to take, given the demand for (and cost of) software developers today? What’s happened?
    JPMorgan’s hiring drive is no mystery. Virtually every company in the world is leveraging software, with many transforming themselves into software organizations to drive new efficiencies, widen profit margins and create new markets. Software has been disruptive, so much so that the current era has been dubbed the “third industrial revolution.” Consumers are benefiting from switching providers at the click of a few buttons and most consume services through software. Bad software experiences today mean rapid customer churn and competitive disadvantage.

    Quality Should be the Heartbeat of any Organization

    In spite of quality’s importance, testing is still perceived as a secondary discipline and a check-the-box exercise. This reflects practices developed for assembly lines during the original industrial revolution. A product gets assembled and testers verify its quality as it leaves the production line. Testing’s goal in this context was to identify problems before they reach end-users. This same approach has been applied widely to software development and is still used by “waterfall” organizations.

    Yet, software development has been revolutionized by new methodologies and practices. Software today is constructed at incredible speed. Notice I’m using the word “constructed” and not “built”—the best developers leverage libraries or APIs to assemble software using existing components. They then add a few pieces of custom business logic to define the intellectual property of an application. Many organizations don’t even code software, instead customizing off-the-shelf software with vast configurations to modify functionality.

    The assembly-line approach to software development is no longer appropriate. When testing is left late, defects are found before a release or even in production. Release cycles are longer and products poorer quality. Yet, many organizations still stick religiously to this approach, with testing teams operating in distinct silos.

    The Role of the Tester is Evolving

    The best organizations in the world are changing how software is built. Testing is no longer a check-the-box exercise at the end of the assembly line. It is transforming into a discipline that revolves around evangelizing and driving quality throughout an organization.

    Quality should be baked in as early as possible, touching every artifact produced. Ideally, this should occur from the start, as a requirement is formulated, all the way through to the software being shipped. The idea of siloed teams working on different pieces is outdated. For quality to thrive, teams must collaborate.

    Modeling offers one approach to driving quality and collaboration throughout an organization. Its journey starts with a requirement. These are typically formulated by business analysts or product owners. Having been a product owner, I found that this usually involves lots of scribbles on paper before communicating to stakeholders on a whiteboard. The preliminary requirement then gets reviewed and refined.

    At some point, the requirement will be converted into walls of text in a project management tool. Modeling intervenes at this stage, adding visualizations that make requirements more consumable, less ambiguous and easier to understand.

    The word “modeling” may seem a scary concept, but it need be no more than a flowchart that encapsulates a requirement. The benefit of modeling is not only that it provides a digital collaboration asset; models can also generate coverage-focused tests before a line of code is written. Modeling creates requirements that are quality-driven and quality-driving.

    The Journey of Quality-Driven Requirements

    • Models Provide Data Upfront

    Developers today require rich data to develop, while testing is only as good as the available test data. Yet, many organizations still feed slices of production data into developers and test cases.
    This approach is problematic for several reasons. First, it often risks legislative non-compliance. Second, production data features large volumes of low-variant data. Data may not then be available when developing unusual-but-important scenarios. Imagine an overdrawn account with several beneficiaries: What should happen when a transfer is attempted?

    Modeling defines data explicitly in requirements. It allows engineers to understand a requirement explicitly, including the relevant data. From the modeled definitions, correct data can then be masked and subset, or generated synthetically. Data can further be delivered on-demand and at scale using database virtualization.

    • Models Provide Understanding

    Once the requirement and necessary data are available, code can be developed. Using text-based requirements, misinterpretations often lead to incorrect functionality. Visual models instead align closely with business logic, and we often see developers enriching models with additional context so that models align with the actual software implementation.

    • Models Generate In-Sprint Tests

    Once the code is developed, the same tests which were created earlier from the requirements can be executed against new functionality. This facilitates reuse and means the test cases are ready to be executed immediately.

    Some tests might further be selected for automation. In this instance, automation can be overlaid onto the requirements model, reusing code to generate scripts. These scripts then become part of the standard regression pack as new features are added to the product.

    A Change Request Comes In

    With iterative development, change requests are common. This typically creates havoc for quality, since it is difficult to understand the impact of a change, and time-consuming to update artifacts like tests and data.

    Traceability is key to overcoming this challenge. Models can connect to external DevOps tools to provide end-to-end traceability, from a requirement, all the way to code and tests. The impact of a change can then be identified directly. Models can be updated, automatically synchronizing test data, test cases, and automation to bring everything up to date.

    Testing for the Fourth Industrial Revolution

    Software is playing a vital role in virtually every organization, as already recognized by organizations thriving today. Baking quality in early is the only way to ensure quality software. The role of testing must change on a large scale. Models offer a way to achieve a “shift left” and collaborative approach to quality while testing earlier and more rigorously by automating and optimizing test and data creation.

  • Adopting Shift Left Testing in Software QA

    Adopting Shift Left Testing in Software QA

    I am often asked to recommend best practices for building software testing programs. The problem is that it depends on your definition of “best.” What works for an innovative startup developing software and deploying agile development sprints is not necessarily going to suit an established public company launching a new web product. The software development industry is so broad and diverse that there is no one-size-fits-all set of practices I can recommend. Put simply, the “best practices” are those that produce consistent results over time, meaning that they can be standardized. There has been a trend toward shift left testing in the past decade, and I have a few suggestions for getting the most out of development teams taking on this new responsibility.

    Shifting Left Testing Supports Agile Software Development

    “Shift left testing” means testing software for functional quality during the development phase, rather than waiting for the QA testing phase within the traditional software development cycle. Back when software was packaged for sale and deployment rather than delivered as a service, there was time in the development cycle for a dedicated QA phase. Developers started with a concept, analyzed their requirements and came up with a design before beginning to write the code. Once written, the software code would be integrated within existing systems (if needed) and then the software would be passed on to the QA team. Any bugs were sent back to development to be fixed and sent through the cycle again. All of these phases represented a conceptual move from left to right.

    Shift left is a departure from this cycle in which developers test their own software in an effort to speed up the process. This kind of development works particularly well for dev teams engaged in agile development. By having developers test their software as they go, bugs can be identified earlier in the development process, which saves time and resources and enables more thorough testing.

    Shift Left is Also Problematic

    There are some significant problems with the Shift left method that can’t be ignored. Specifically, developers are being asked to perform functions they were never properly trained to do—and often have no real interest in doing.

    Software developers are creative people by nature. Whether self-taught or formally trained, the majority of them have come to their positions in pursuit of creating something. Quality assurance testing, by contrast, is a destructive process. The goal of the QA engineer is to break things—kicking the tires and carefully documenting their destruction. And when you ask your creators to become destroyers, it doesn’t always go well.

    We can’t fault developers for not wanting to find faults in their code. Just as the best builders are not the best demolition experts, software developers want to create and then move on from one project to the next. So their reluctance to engage in testing is understandable. It’s part of who they are. The truth is that most developers find testing a monotonous chore. We need to learn how to encourage and incentivize them to become good testers of their own code.

    Provide an Encouraging Environment

    The first step toward making developers better testers is to educate them on the importance of testing as part of the development process. They may initially think that you’re only interested in saving money on hiring QA testers, or that they are being asked to perform the work of two employees. Introduce shift left testing as an integral part of the development process, explaining that continuous early testing is key to building high-quality software. By putting testing within the context of elevating software quality, developers will become more engaged in the process, raising the stakes and their involvement in projects.

    Make Software Testing Easier for Your Developers

    Since testing is not likely to be the primary interest of your developers, simplifying the task is going to make it at least more palatable to them. Test automation tools can be a big part of this strategy. Automated test scripts can be run flawlessly, frequently, and with much greater load capacity than with manual testing. You’ll be able to scale your tests to mimic a single user, or thousands of concurrent users, to discover how your software holds up under high demand. Pixel-for-pixel aberrations and errors in your graphic user interface (GUI) can be analyzed more quickly and effectively than with manual testing.

    Automated testing also takes much of the monotony out of testing, preventing burnout and resentment among your employees. Writing scripts also has the benefit of being, inherently, a creative, not destructive, process. By integrating the running of scripts into the normal workday of the developer, and emphasizing their value in building the best software possible, testing becomes a valued task, rather than a mindless chore.

    Providing Other Incentives

    When you adopt shift left testing, you just might be asking your developers to take on a role that they don’t enjoy as much as coding. And they may never enjoy it as much. But that doesn’t mean there aren’t actions you can take. If you want to keep your most valued developers happy and encourage them to stay with you, think about providing other incentives that make the entire job package irresistible. Make your company a great company to work for. From the working environment to other perks and compensation, any way you make your team happy can help them manage integrating new responsibilities.

    Testing Tools Lighten the Load

    As test automation technology gets more sophisticated, testing tools that incorporate aspects of artificial intelligence technologies to assist in test script generation and self-repair will play a larger role in enabling your developers to focus on the more creative aspects of the job — addressing design requirements and functionality (the “what” and “why” factors) in novel ways, and leaving more analytical, procedural tasks (the “how” factor) up to the tools. As testing tools perform more of the monotonous tasks for us, and with greater accuracy, developers will have more time for the creative tasks that keep them happy. These technologies are evolving as we speak and will address many of the issues described above. But you might still want to consider buying some doughnuts for the break room.

  • Three Ways to Speed up SAST

    Three Ways to Speed up SAST

    In modern, continuous software development life cycle (SDLC) processes, when code is written and before it’s committed to the repository, it’s run through testing, which may include unit testing, regression testing or static application security testing (SAST). The benefit of SAST for DevSecOps is the real-time feedback it provides developers as they create and make changes to the source code before they submit it. The amount of time that’s acceptable in this kind of workflow is 15 minutes to 30 minutes maximum, with some allowance for how big the change is. Fast feedback means quicker turnaround and more progress on features. The business value lies in those features. Testing makes sure they work.

    Depth Versus Breadth

    Checkers used by SAST are not all created equal, at least in terms of the computation power needed. The time for analysis is directly related to the complexity of the checkers used to detect certain types of vulnerabilities. Typically, coding standard enforcement rules are easier to compute versus complex tainted data analysis used to detect command or SQL injections, for example. Therefore, there is a trade-off to be made in the types of bugs and vulnerabilities being detected. When developers are coding, it makes sense to optimize the compute time for SAST to reduce any delays. During software builds, more time and compute power is available so depth and breadth can be increased to catch complex vulnerabilities.

    Compute Power

    SAST analysis time scales well with compute power. The availability of more computing resources such as CPU and memory, can impact the time of analysis. Bigger is better in this case and investing in additional hardware for SAST can pay off in terms of productivity.

    Codebase Size

    SAST analysis time is also dependent on the amount of source code analyzed. Depending on the application, the code base is a determining factor in terms of what depth and breadth of analysis is optimal.

    There is some correlation between small code base size and strict requirements for SAST such as in safety-critical software. It’s expensive to build and test safety-critical code, but code size is typically smaller than in other types of applications. However, these products need to conform to industrial coding standards and thus require rigorous levels of testing and analysis to detect a wide variety of defects.

    Larger and extremely large code bases require a trade-off in terms of depth and breadth of analysis. Prioritizing the types of defects and vulnerabilities to concentrate on is critical.

    Speeding up SAST means reducing the amount of work. The most intensive operation is a full analysis of the entire source code base. Just as full compilation from scratch takes a long time, the same is true of SAST analysis.

    To accelerate SAST processing times, follow the same three techniques used to avoid compilation in large C/C++ projects: incremental analysis, component analysis and developer analysis.

    1. Incremental Analysis

    Incremental builds are a major factor in reducing developer build times for C and C++ projects. Small changes don’t require the entire code base to be recompiled. It’s impractical and unnecessary. When a developer modifies something, only what’s needed to be recompiled gets recompiled automatically, and the build infrastructure handles it.

    SAST works in a similar manner. As source code is being compiled, SAST parses the same source code and creates a program model. The second phase is analyzing that program model against the configured checkers and generating warnings. The warnings typically go into a database for later analysis.

    During the build of the software, SAST is run in parallel. Although similar to a compiler in operation, SAST must do more work than the compiler and takes longer to finish and provide results. The amount of code needed to recompile for each change directly impacts analysis time.

    Meanwhile, SAST of an incremental build requires a lot more work than just analyzing the changes. SAST uses abstract execution of code flow through multiple different compilation units. So even though a single file is modified, the calls it makes to other files need to be analyzed as well. The incremental analysis doesn’t increase the build and parse workload, but the analysis scope will be a lot bigger than expected. So, whenever you do an incremental build, SAST will need to reanalyze the dependencies, changes and related bits and pieces around those changed units.

    Nevertheless, it remains a good option for reducing SAST analysis times.

    2. Component Analysis

    If the software being developed has a loosely coupled, component-based architecture with well-defined interfaces, it’s possible to isolate both the testing and the SAST analysis to the component where changes were made. Just as this type of architecture simplifies many aspects of development, it pays off with SAST as well. It’s possible to analyze the component in isolation and get good results.

    3. Developer Analysis

    To further reduce SAST workload, start the first build without SAST, which removes the burden of the extra computation needed. Since subsequent builds are a lot smaller and build speed is much faster.

    However, there is an increase in false negatives due to the smaller scope of analysis since only files that changed are analyzed. This smaller scope means that all possible dependencies are outside the program model and are not considered in the analysis. However, this rapid analysis works well with coding standard enforcement, for example.

    Developer SAST

    Where to Start?

    Begin with a full build and analysis. This sets a baseline for detected bugs and vulnerabilities and establishes a performance target for reducing build time. This step will determine the exact number of lines of code, how long the analysis takes and its impact on build server load. These metrics should drive the analysis optimization effort.

    Given the options, the first step should be a component-based analysis. If this works well, it’s still possible to further reduce analysis with incremental and developer analysis within components. This approach can be integrated into typical developer workflows.

    When creating a developer branch for implementing changes, quick developer local analysis will catch local errors. These tend to be the largest set since they represent the largest part of the code changes. When enforcing coding standards, this is the point where violations are identified. A full analysis is recommended on a merge request to detect remaining defects related to cross-component-boundary behavior.

    Speed up SAST

    To speed up SAST consider using these techniques at the appropriate development stage to reduce time to complete, while still producing excellent results. Keep in mind that:
    ● Complete analysis yields the best and most precise results,
    ● Incremental analysis only analyzes recompiled code that has been modified,
    ● Component analysis works well in software that follows a well-defined component architecture and
    ● Developer (local) analysis provides a good compromise of precision and analysis time.

  • Four Causes of Technical Debt in DevOps

    Four Causes of Technical Debt in DevOps

    Ideally, DevOps should retain a lean footprint, but avoiding technical debt is easier said than done. As such, over half of IT leaders report technical debt is a big or critical problem. Without routinely addressing technical debt, DevOps teams can easily face inconsistencies during deployments. Versioning can get out of hand without consistent upgrades and aging legacy components can be a roadblock to development fluidity.

    Simply put, letting too much technical debt pile up isn’t all that great.

    I recently met with Kurt Andersen, SRE architect at Blameless, and Matt Davis, SRE advocate, to gather insights on the sources of technical debt in the field of DevOps. According to Davis, “Tech debt is something that you really want to keep chipping away at.” Below, we’ll highlight some common causes of technical debt and suggest ongoing practices to help tame it.

    1. UI-Based Deployment Patterns

    UIs are great for quickly configuring DevOps, but if your deploys aren’t codified, this reliance on UIs could come back to haunt you in the form of technical debt. A lot of the time, explained Davis, infrastructure teams have a tendency to just use the GUI of their given cloud provider to initiate deployments. Perhaps the team is wrapped up in feature velocity or a deadline to release.

    For example, you could use a UI to quickly create a database, but this is likely not a repeatable and robust process. Especially as an organization scales its cloud adoption and uses multiple clouds, you start to produce “snowflake” deployment patterns across teams, says Andersen. This could lead to inconsistencies around different deployment options per region.

    It’s not easy to catalog a configuration if it’s not written in code, and the more undocumented UI-based deployment patterns are used, the more silos of knowledge you create. “If not codified with Terraform, you have to document what you did—code gives you that documentation,” said Davis.

    Solution: Rely less on the GUI and codify your deployment patterns with infrastructure-as-code (IaC).

    2. No End-of-Service Life Cycle

    Development is typically more concerned with feature velocity than planning for service deprecation. Yet not envisioning the entire life cycle or setting an off-ramp from the start can produce more technical debt later on. “No one pays attention to planning for retirement, and at the end-of-life, services are hard to retire,” described Andersen.

    What you end up with are these “senile services” that aren’t doing that much but remain critical to the business’s operations. These services may be challenging to migrate. Or, they may be the product of unknown shadow or zombie APIs. For example, a marquee customer may depend upon a certain legacy endpoint, making the service difficult to evolve.

    Not planning for the end of the service lifecycle can hold up your development from more efficient ways to work. For example, if one “senile service” has never used a particular commit strategy before, it could stall agility when revamping the deployment pipelines, says Andersen. Thus, it’s best practice to set clear deprecation timelines and plan for the future.

    Solution: Plan the end-of-service life cycle from the upstart.

    3. Lengthy Dual-Track IT

    Another area that can produce technical debt in DevOps is dual-track IT. This is when engineering teams maintain one core stack in addition to an experimental structure and shift toward adopting the experimental structure over time. Development is constantly overseeing migrations to new technologies, yet they rarely plan for the coexistence of both stacks.

    While splitting an old and new stack into distinct units sounds like a temporary measure, some of these large transitions simply take longer than you think. For example, SoundCloud’s journey to refactor its monolith into microservices was an eight-year process.

    Solution: Realize dual-track IT might take longer than you think and plan accordingly.

    4. Intermittent Version Upgrades

    Third-party software is constantly upgrading to new versions. Especially if you’re integrating many open source dependencies into your stack, you’ll find the cadence of version upgrades to be quite fast. These upgrades are often imperative to plug new vulnerabilities as they emerge in the software supply chain.

    But upgrading versions and base images and having a patch management scheme that works from environment to environment requires a lot of work, Davis cautioned. Kubernetes version upgrades, for example, can take a lot of time. Yet, if your upgrades are not matching the pace of change, added Andersen, they can quickly pile up and leave you left behind with a mountain of work. “If you’re not upgrading monthly, you’re in danger.”

    Solution: Have a continual upgrade process in place.

    Technical Debt: Keep Paying, Cleaning, and Pruning

    Technical debt is something you need to grapple with before it gets out of control. But it’s not always apparent where it lives. Looking toward the future, automated tech debt scanning could help discover where technical debt is and provide remediation points as part of ongoing software stack auditing.

    You don’t want to get to the point where you accumulate and accumulate technical debt, only to burn it all to the ground in one disastrous fell swoop, said Davis. Similar to the way you slowly pay off a mortgage each month, technical debt is something you must continually keep an eye on and make reducing part of your team’s continuous workflow.

    Or, consider the old camping mantra of always leaving the campground cleaner than you found it. You pick up a little bit of extra trash and take it out with you, making the environment better for everyone. Or, consider how a gardener must prune the branches of a fruit tree to help it grow. Pick the analogy that works for you and see if you can apply it to continually reduce tech debt in your DevOps processes!

  • Sonatype Report Surfaces Scope of Known Vulnerability Challenge

    Sonatype Report Surfaces Scope of Known Vulnerability Challenge

    Sonatype this week published a State of the Software Supply Chain Report that found a 633% year-over-year increase in malicious attacks aimed at open source software residing in public repositories.

    In addition, Sonatype launched a Sonatype Safety Rating system that employs machine learning algorithms and other metrics to identify the most secure open source components stored in those repositories.

    The report found 96% of open source Java downloads have known vulnerabilities that could have been avoided if an available, more secure version had been used. Collectively, the report estimated there were 1.2 billion known-vulnerable dependencies that could be avoided if developers had only been aware that a later version of a software component was available.

    According to the report, the average Java application contains 148 dependencies. Updates to Java applications are, on average, made 10 times a year, so development teams are being asked to track 1,500 dependency changes per year.

    Sonatype CTO Brian Fox said the report makes it clear that organizations are not assuming enough responsibility for using open source software that has known vulnerabilities. In effect, organizations are blindly and recklessly using software components without any real appreciation of the associated risks, he said.

    Rather than a knee-jerk reaction to each new zero-day vulnerability that is discovered, Fox said, IT teams need to focus on putting processes in place to make sure the latest, most secure edition of any given software module is being used. A survey of 662 engineering professionals included with the report, however, found 68% of respondents were confident that their applications were not using known vulnerable libraries. Nevertheless, a random sample of enterprise applications conducted by Sonatype found 68% contained known vulnerabilities.

    Overall, the report surmised that consumption of open source software will soon surge past an estimated 3.1 trillion total requests, with six out of every seven vulnerabilities being the result of a transitive dependency created by indirect relationships between software components. At the current rate of consumption, it’s fairly clear that the processes used to build and deploy applications today are insane given how dependent organizations are on applications, said Fox.

    In the wake of a series of high-profile security breaches, there is, fortunately, a lot more focus on securing software supply chains today than there was this time last year. The scope of the challenge, however, is mind-boggling when you consider the totality of applications already deployed in application environments that have known vulnerabilities. Organizations need to first make sure they are remediating vulnerabilities in their most critical applications while at the same time implementing DevSecOps best practices. This will ensure as few vulnerabilities as possible find their way into the next generation of applications.

    Obviously, these application security issues are not going to be resolved overnight. DevOps teams will be spending the next several years addressing a level of technical security debt that has been allowed to accrue for more years than anyone in IT really cares to admit.

  • Datadog Extends Reach of Integrated DevOps Platform

    Datadog Extends Reach of Integrated DevOps Platform

    At its Dash 2022 conference, Datadog announced today it is extending the reach of its namesake cloud-delivered monitoring and observability platform to address continuous testing, application security and cost management.

    In addition, Datadog has made available in beta a Data Streams Monitoring tool that makes it simpler to identify upstream issues that are likely to impact DevOps workflows. Also in beta is a Dynamic Instrumentation tool that makes it possible to employ Datadog agent software to collect data on the fly from specific source code to investigate performance issues. Datadog is also making available in beta a Resource Catalog to make it easier to navigate cloud resources that IT teams are employing and has added a Datadog Workflows tool to create blueprints to for automating repeatable processes.

    Datadog is also expanding the reach of its Watchdog telemetry analysis service infused with machine learning algorithms to add beta support for containers. There is also now a beta of an Event Management tool that DevOps teams can employ to more easily investigate a stream of related alerts using the Watchdog service.

    The Datadog Continuous Testing service, now generally available, provides DevOps teams with a workbench for generating and maintaining tests that can be integrated within continuous integration (CI) workflows. Datadog has also added an Intelligent Test Runner tool, in beta, to enable DevOps teams to only run tests that need to run when a change is made by analyzing code execution paths to increase developer productivity.

    Renaud Boutet, senior vice president of product at Datadog, said the goal is to make it possible for DevOps teams to create and run tests from within the Datadog user interface in parallel using a set of no-code tools without having to employ scripts. The capability will make it possible to automatically generate tests using data collected by the Datadog platform as part of an infinite DevOps workflow loop, he added.

    At the same time, Datadog is making generally available a cloud security management service that combines its existing cloud security posture management (CSPM) service and the cloud workload security (CWS) service to provide an integrated cloud-native application platform (CNAP) at no additional cost. In addition, Datadog is making it possible to block access to application components that are under attack or might be attacked in real-time based on the known vulnerabilities detected using tools that are currently in beta.

    That approach reduces the total cost of cybersecurity by providing IT teams with a single service through which they can manage application security using a platform Datadog gained last year with the acquisition of Sqreen.

    Finally, Datadog is making generally available a cloud cost management tool that leverages its monitoring capabilities to enable organizations to automatically attribute spend to applications, services and individual teams. Changes to spending patterns that impact costs are surfaced alongside the same tools IT teams are already using to monitor their cloud computing environment. That unified visibility increases awareness of costs in a way that encourages DevOps teams to optimize the utilization of multiple classes of cloud resources.

    The services being added are extensions of a Datadog platform that employs a single set of agents it provides. The Datadog platform also supports the open source OpenTelemetry agent to deliver a wide range of services. That approach eliminates the need for IT teams to deploy multiple agents each time they add an additional monitoring, security, testing or cost management tool.

    Datadog has been making a case for an integrated platform that reduces the level of integration effort DevOps teams would otherwise need to expend when deploying separate point products to address monitoring, observability, security and cost management issues. That approach makes it easier for DevOps teams to collaborate and reduces the total cost of DevOps. In contrast, each time a DevOps teams adds a point product to address a specific issue, they need to deploy and maintain a separate set of agents.

    In the longer term, it’s not clear how problematic that agent issue will be once open source OpenTelemetry agent software is more widely employed. However, it may still be years before open source software provides the same capabilities that are available today using proprietary agent software.

    It may take some time before every application is as fully instrumented as it should be, but as monitoring and observability platforms continue to evolve, it is possible to more proactively manage application environments.

  • Making SBOMs Actionable

    Making SBOMs Actionable

    A software bill of materials (SBOM) is a list of all the software components found in a given codebase or used in a given software build. Great. So, now what? Why do we even care about SBOMs?

    Those are great questions—because in and of itself, the SBOM doesn’t really do anything; it is simply a means to an end. That end being greater software security and a more secure software supply chain.

    Let’s take one step back to understand why there is such a sudden and urgent demand for SBOMs. The most recent ‘big bang’ attack on the software supply chain was the Apache Log4j flaw, discovered in December 2021. That vulnerability opened the doors to countless breaches and caused billions of dollars in damage. Before that, the high-profile hacking of SolarWinds caused similar damage.

    Now, will the SBOM prevent that from happening (again)? No. However, what an SBOM can do is tell you exactly where these kinds of vulnerabilities exist in software and enable you to quickly patch your systems or block exploits. Having a software bill of materials enables you to quickly assess the risks in your codebase and mitigate them as needed.

    It’s strongly recommended that you use dynamic SBOMs (versus static) that are automatically updated to the latest SBOM information and provide extensive search capabilities as part of an SBOM database. That allows for immediate identification of a specific dependency version, like the Log4j case. Some solutions also update the SBOM information based on the runtime (like Docker or Kubernetes). That way, you know not only if some component is used, but also where.

    SBOMs are an invaluable tool as they allow an audit of licenses, libraries, modules, applied patches and other components to look for weaknesses. However, for such an audit to reliably trace the components used to build the software, the SBOM must have certain attributes.

    • Unique identities for each software component
    • Separate identities (independent of those used internally in the organization) to identify each machine and user involved in the development process
    • Timestamps to facilitate traceability of each change or component incorporation

    Additionally, from a security point of view, a key element necessary for the reliability of the SBOM is to prevent unauthorized changes to it. And the most effective way to do this is by using an immutable ledger that records a history of when each change was made.

    Once the reliability of an SBOM is assured, DevSecOps teams can use it as part of their threat scanning toolbox and, therefore, improve software security.

    There is no doubt that SBOMs should be requested from your software vendors and that you should consider creating SBOMs along with your own developed software. It’s all about the proper storage of the SBOMs so you can be sure they’re recent, searchable and trustworthy and tamper-proof.

    The benefits and use cases for SBOMs are numerous; they vary across stakeholders who produce, choose and operate software and are amplified when combined. Use cases for SBOMs include better software development, supply chain management, vulnerability management, asset management and high assurance processes. The benefits include reducing cost, mitigating security risk, license risk and compliance risk.

    But the key is making the SBOM actionable.

    No developer, no software maintainer or DevOps engineer wants to manually collect the dependencies and produce SBOM documents. It needs to be fully automated within the software build and deployment pipeline and there needs to be a proactive check of where it’s currently running.

    The most efficient approach to work with a pervasive SBOM policy is to have them generated as a byproduct of a modern development CI/CD process.

    In today’s complex development environments, it is imperative that SBOMs can be shared frictionlessly between teams and organizations as a fundamental part of software management transparency. This transparency model is supported by the ISO association through its OpenChain Specification, and it is part of an effort to improve security in the digital industry.

    There are two SBOM specifications: The Software Package Data Exchange (SPDX) backed by the Linux Foundation and the CycloneDX specification managed by the CycloneDX Core working group. For more details, have a look here.

    Make sure to choose the SBOM standard that makes the most sense for you and stick to that for all of the software developed or used in your infrastructure. The test of whether your SBOM is actionable will come when the next Log4j moment arrives. You should be able to pinpoint the existence and the location of the component in question to mitigate risk effectively.


    To hear more about cloud-native topics, join the Cloud Native Computing Foundation and the cloud-native community at KubeCon+CloudNativeCon North America 2022 – October 24-28, 2022