devopsy.kinsta.cloud

Category: Continuous Testing

Continuous testing is the process of executing automated tests as part of the software delivery pipeline to obtain immediate feedback on the business risks associated with a software release candidate.

  • Why Your DevSecOps Initiative Will Fail

    Why Your DevSecOps Initiative Will Fail

    When you’ve been in and around the security industry for long enough, you get used to the industry hype machine turning a cool innovation into, uh, meh. This hype cycle starts at the RSA conference each year, and folks like me look for new hot stuff on the show floor. For perhaps only the second time in 25 years, I wasn’t at the show in person, but it seems that DevSecOps was the preferred bandwagon on which every vendor jumped. Sigh.

    I’m not sure whether it’s my increasing age or this business that has made me cynical, but here we are.

    To be clear, from a customer’s perspective, there is great value in the concept of DevSecOps, as evidenced by the upcoming DevOps Connect: DevSecOps @ RSAC 2022 virtual event. Having these (sometimes) mortal enemies working together toward a common goal is great, in theory. But alas, in practice (and similar to what happened when DevOps hit the scene), it’s easier said (or displayed in a vendor booth) than done.

    Thus, I’ve come to a sobering conclusion: Your DevSecOps initiative will fail.

    Most organizations will talk a good game for a while and they may even try some stuff, buy some kit, send folks to training and the like. But in the end, despite all these efforts, you guessed it—they will still fail.

    Is this a foregone conclusion for every organization? Well, no. But this isn’t my first rodeo, and I’m pretty sure this one will end like the others—in failure.

    But let’s say you want it to be different this time. Really, you do. And you are willing to take some actions to give yourself a chance at success. If that’s the case, here are a couple of tips to increase your chances, however small.

    Culture

    First and foremost, DevSecOps is not a thing. It’s not a set of tools (no matter what the vendors tell you); you don’t buy it on a VAR line card. DevSecOps (like DevOps) is a cultural shift. Your teams need to work together. They need to collaborate. They need to have shared incentives. And they need to experience the consequences of not playing nicely in the sandbox.

    Cultural shifts are hard. People don’t like to change. They resist. They undermine. They sabotage. It all sounds nefarious, but they do. So unless you can change your culture, you’ve got no shot.

    Shift Left

    Another one of these security-marketing-terms-run-wild is shift left. It means you want to integrate security testing directly into the development process as early as possible. You should be running security checks in the developer’s IDE to ensure they don’t violate security policy (like hardcoding API keys in their source, for example) before they even commit the code.

    You must also integrate security testing into the CI/CD pipeline(s). Thus, you need tooling that works with your preferred pipeline.

    Security Champions

    Security testing will create angst for your developers. They will have false positives. They will have to fix things that they probably don’t understand. It will cause them to miss delivery dates. Of course, they should build it right the first time, but in their mind, they’ll see it as the security folks causing delays.

    The developers will be annoyed, and you’ve got to manage that annoyance—or have senior management quash any rebellions, but that’s not a long-term solution.

    On the other hand, you could establish a security champions program, which anoints (and trains) developers on each team to “represent” security within the development team. They help educate the developers on the tools and secure coding. They diffuse irritation when something goes wrong.

    A strong security champions program is one of the leading indicators of DevSecOps success.

    Break the Build

    Finally, the security team must have the juice to break the build if a critical issue is found during the deployment. We talk about this with many organizations and they nod their heads. But they are lying to me and to themselves. When it comes down to it, someone on the dev team calls someone in senior management and, all of a sudden, that critical issue is now an alert, not a failed deployment.

    Not being able to stop a build will kill your DevSecOps program. It’s like when a quality control issue forces someone to stop the assembly line. If you don’t stop the line, faulty products can result. In a security context, that means some critical vulnerabilities will be much more expensive to fix later.

    So, there you have it. If you can evolve your culture, integrate security testing into your toolchain, empower security champions and empower your team to stop a deployment, you’ve got a chance at a successful DevSecOps program.

    Unfortunately, you probably can’t do these things, and you’ll likely fail. In that case, I have some lovely property by a waterfall I’d like to sell you.

  • Survey Shows Increased Reliance on DORA Metrics

    Survey Shows Increased Reliance on DORA Metrics

    A new survey found just under a quarter (23%) of respondents are now tracking all four of the DevOps metrics defined by the DevOps Research and Assessment Team (DORA), with another 17% now tracking three.

    The State of Developer Experience survey polled 129 IT professionals that play a role in software development and was conducted by LeanIX, a provider of a platform for managing the building of enterprise applications.

    Dominick Rose, vice president of product management and strategy for LeanIX, said the ability to track DORA metrics such as deployment frequency (77%), failure rate (73%), lead time for changes (63%) and mean-time-to-recovery (MTTR) (55%) are leading indicators of DevOps maturity.

    DORA metrics, which are defined and maintained by a unit of Google, are being used most frequently to drive agile planning (69%) and optimize continuous integration/continuous delivery (CI/CD) platforms (58%), the survey found.

    Overall, the survey revealed just under half of the respondents (47%) said their organization had a high level of DevOps maturity, defined as having adopted three or more DevOps working methods. Those working methods are: Being flexible to changes in customer needs; having implemented a CI/CD platform; all engineers build, ship and own their own code; teams are organized around topologies and each team is free to choose its own technology stack.

    Of course, each individual organization will determine for itself what level of DevOps depth is required. For example, not every organization would see the need for teams to be organized around topologies or be free to choose its own technology stack. In fact, Rose said the survey made it clear that larger enterprise IT organizations tended to have a lower overall level of DevOps maturity. One reason for that is many larger organizations are still employing legacy processes to build and deploy software, noted Rose.

    Most developers are also further along in terms of embracing continuous integration (CI) than IT operations teams are in adopting continuous delivery (CD), added Rose. However, as the number of applications being developed using low-code and no-code tools continues to increase, it’s only a matter of time before application delivery needs to become more automated, he said. Achieving that goal will become easier as more artificial intelligence (AI) is incorporated into IT operations, Rose added.

    Nevertheless, the survey made it clear there is still a long way to go in terms of bridging the divide between DevOps teams and the rest of the business, with only 20% of respondents indicating that they use a value stream management (VSM) platform to track engineering efficiency to tie development activities to actual business results.

    A total of 71% of respondents, however, tracked open support tickets as a measure of customer value, followed by 66% that tracked monthly users and 45% that tracked feature adoption.

    It remains to be seen how DevOps will be further promulgated throughout organizations but as more organizations realize how dependent they are on software, it then becomes a question of how fully they embrace DevOps best practices as the pace at which applications are being built and deployed continues to accelerate.

  • Buildkite Adds Analytics Tools to Identify Flaky App Tests

    Buildkite Adds Analytics Tools to Identify Flaky App Tests

    Buildkite has added an analytics tool to its continuous integration/continuous delivery (CI/CD) platform that identifies flaky tests.

    Buildkite’s co-CEO Keith Pitt said the company’s Test Analytics tool enables continuous performance monitoring and real-time insights for test suites that, when not properly written, only serve to waste DevOps teams’ time. Rather than eliminating those tests, most DevOps teams just opt to run them again on the assumption that the code they are testing will eventually pass because the test is poorly constructed, noted Pitt.

    Buildkite reports there are more than 800 million seconds—or the equivalent of 25 years—of flaky tests manually re-run on its platform by developers each month.

    In addition to being integrated with the Buildkite CI/CD platform, the Test Analytics tool can also be applied to tests residing in CI platforms such as CircleCI, GitHub Actions and Jenkins.

    Test Analytics includes a dashboard that identifies the slowest and least reliable tests to help prioritize test remediation efforts. Historical views and trend graphs are also available to provide further insight into recurring issues.

    DevOps teams can also take advantage of user-defined thresholds to define whether a test is slow or unreliable, which generate custom alerts when discovered. Finally, there are integrations with commonly used testing frameworks to determine where tests are, for example, wasting time running database queries against application programming interfaces (APIs) that have timed out.

    Pitt noted that most tests today are created by developers and software engineers rather than dedicated testing teams. As such, the chances there will be an issue with a test that is returning flaky results are high.

    Organizations are, of course, trying to determine how far left to shift application testing. Developers can write their own tests, but how thorough and objective those tests are is debatable. There are also certain tests that should be applied to all applications that organizations can embed within their DevOps workflows to make sure they are always run.

    Obviously, the more testing that occurs the fewer issues DevOps teams will encounter after an application is deployed. However, when tests are flaky, it becomes easier for developers to justify not running them even though that test might still surface an issue they should address. In addition to increasing confidence in the quality of those tests, knowing which tests are flaky allows a DevOps team to determine which tests should be run first. That’s critical at a time when more organizations are trying to increase developer productivity; that productivity is often adversely impacted by waiting for tests to be completed before more code can be written.

    Eventually, the overall quality of tests should improve as machine learning algorithms and other forms of data science are applied. In the meantime, however, the onus for improving application testing is clearly on the engineers that create them. The challenge is many of those engineers may not even be aware how flaky a test really is until someone shows them some empirical data that can’t be disputed.

  • The Two Types of Code Vulnerabilities

    The Two Types of Code Vulnerabilities

    Spell checkers, grammar checkers and predictive keyboards all help reduce errors in written communication, but in a creative medium where writers need to innovate new expressions, nothing can eliminate all errors. The same is true in the code we write.

    Of course, your code is probably perfect but, on average, all code has bugs. Some of those bugs are benign. Some will cause unexpected early termination. A few will cause runaway resource consumption. And some bugs lead to security vulnerabilities that let users do things that are unexpected and sometimes undesirable.

    Those security bugs are behavioral vulnerabilities because they depend on how the code behaves when executed. Another type is an intelligence vulnerability, something that’s part of the correct behavior of the code, but which gives away details about how to exploit a system. These are the two main types of code vulnerabilities; both come with risks.

    Behavioral Vulnerabilities at Runtime

    Tl;dr: How the code works and how attackers can leverage what it does to their advantage.

    This covers a broad range of problems, but all of them are a result of the behavior of the code leading to undesirable outcomes in specific circumstances. Most of these are unintentional. An unescaped user input sent to a database can lead to a SQL injection vulnerability or an unsanitized string sent to a browser could create a cross-site scripting (XSS) vulnerability. Some behaviors are intended, but have unexpected consequences; the Log4j string substitution behavior that allowed it to output environment variables or read data from external resources was intentional, but the malicious usage we now call Log4Shell was not. In another recent example, a combination of behaviors could be exploited to gain root privileges on Linux desktop computers.

    Intelligence Vulnerabilities in the Content

    Tl;dr: What the content of the code tells an attacker about the system and how to access it. 

    In the 1950s, the phone company—there was pretty much just one at the time—began implementing completely automated dialing that didn’t require a human operator at the switching station to connect the call. The details were first published in a technical journal in November 1954 (see pages 1209 and 1309 here), and the information gleaned from that led to the creation of “blue boxes” that used insider knowledge to allow outsiders to control telephone equipment and make free long-distance calls.

    That’s hardly the first example of insider knowledge being leveraged for outside advantage, but its role at the heart of computer hacking and the origins of the PC (the ‘hack’ inspired Wozniak and Jobs) is so legendary that it demands recognition here.

    The phone system’s intelligence vulnerabilities were published in the company’s technical manuals—manuals that many thought were only for internal use, and nobody realized were available to untrusted readers. Today, though, intelligence vulnerabilities are in our code.

    Secrets abound in code, along with the details about which systems those secrets connect to. These secrets were once difficult to manage and putting them in code was the only practical choice. Now we have tools like Hashicorp Vault, as one example, and secure keystores in almost every cloud, but the legacy and security debt of all those secrets in legacy code are a goldmine for attackers like Lapsus$ and others who specifically target code.

    Security Is a Process, not a Product

    There’s a lot hidden under the surface in the two categories outlined here. Each of us can name many types of behavioral vulnerabilities—many more than the different types of intelligence vulnerabilities—but the two classes of vulnerabilities represent relatively equal risk. Recent events have demonstrated how an attacker with inside knowledge gleaned from code—and with passwords and keys taken from it—can move quickly despite firewalls and other access controls.

    Identifying secrets and other intelligence risks in code is relatively easy, but few teams do it because detection is the easy part. Integrating detection in developer workflows so they can take action in-context while working on and reviewing code is the most effective way to eliminate creating new security debt. And tools that give developers the information they need to understand the overall picture of intelligence risks in their code, the areas of highest risk—and that point them to actionable issues they can fix immediately to improve their code health—are necessary to help teams mitigate historical risks, ideally by rotating any secrets found.

    Tools exist that can automate that process and help teams to eliminate more than half the risk in their code—but the first step is recognizing where that half resides.

  • GitLab To Embed Observability in CI/CD Platform

    GitLab To Embed Observability in CI/CD Platform

    The latest version of the GitLab continuous integration/continuous delivery (CI/CD) platform added a range of monitoring and visualization capabilities as part of an effort to embed a full set of observability capabilities in the core platform within the coming year.

    As part of that effort, the latest version of GitLab, 15.1, has now added support for all the metrics defined by the DevOps Research and Assessment (DORA) team that is also part of Google.

    In total, the 15.1 version of the GitLab platform has more than 30 updates. These include additional security capabilities in the form of support for the security assertion markup language (SAML) Group Sync and the Supply chain Levels for Software Artifacts (SLSA) framework defined by Google.

    Finally, GitLab added links to all configuration files and templates included in the CI/CD platform to the pipeline editor to make it easier to manage large and complex pipelines.

    Cindy Blake, director of product marketing for GitLab, said the monitoring, observability and security capabilities added to the platform are part of an ongoing effort to eliminate the ‘digital duct tape’ that many DevOps teams currently rely on to extend a CI/CD platform. In fact, as many DevOps teams look to better secure their software supply chains, many are now taking advantage of the moment to replace do-it-yourself (DIY) approaches with an integrated CI/CD platform based on a common codebase, noted Blake.

    As part of that effort, Blake said GitLab is building its own dynamic application security testing (DAST) tools that are more tightly integrated with its CI/CD platform.

    In general, there is now an expectation that the tools required to secure software supply chains should be made available within a CI/CD platform, but organizations are still working through the processes that will need to be implemented to achieve that goal, added Blake.

    It’s not clear to what degree organizations are looking to replace legacy CI/CD platforms as they modernize application development and deployment. In many cases, however, the technical debt associated with maintaining custom integrations across their DevOps environment has become considerable. Many organizations are revisiting their DevOps workflows as part of an effort to increase overall developer productivity. The challenge, of course, is that migrating from one CI/CD platform to another requires considerable time and effort.

    In the meantime, the number of organizations embracing DevOps best practices continues to grow. GitLab is betting that as the total size of the CI/CD market continues to expand, most of those organizations will prefer an integrated platform rather than writing and maintaining custom integrations themselves.

    Regardless of approach, the ability to monitor and observe workflows is an essential element of any DevOps workflow. There is, of course, no shortage of observability platforms available. It’s not clear whether IT organizations will adopt one of those offerings versus taking advantage of observability and monitoring embedded within a CI/CD platform. One way or another, however, it’s getting easier to discover the root cause of any IT issue.

  • Creating Career Opportunities in Quality Engineering

    Creating Career Opportunities in Quality Engineering

    Software testing is the thread that connects code to customers, ensuring that the code created by development teams works for users and supports customer loyalty. But as the demands of software development are transformed by DevOps and customer expectations rise, the career and role of the software tester is evolving rapidly. Customer expectations have made it imperative for applications to work seamlessly, necessitating evolution in QA and testing.

    As the QA and software testing sector progress—and “quality engineering” becomes the de facto term for creating agile, customer-centric software teams—QE is attracting a more diverse pool of leaders as the career skillset widens. The more quality engineering becomes the differentiator between elite software teams that thrive in a DevOps world and teams that struggle to adapt, the more important it becomes to elevate diverse leaders in the industry.

    Understanding this, mabl recently held a panel discussion with QA and software testing industry veterans who discussed the new era of quality and how these changes are transforming who enters the field.

    The Evolving Role of the Software Tester: Building Professional Confidence

    The constant evolution in software testing and quality engineering, though exciting, also presents a challenge to QA professionals: How to consistently build the necessary skills as well as the confidence to succeed. As quality engineering becomes a multidisciplinary career role that encompasses engineering, functional and non-functional quality management and the customer experience, the question for QE professionals becomes one of continuous learning and the confidence to broach, if not lead, challenging discussions.

    LaDeitra Lee, chief technology officer for Sparx Studio, understands how confidence can sometimes feel elusive—especially for younger pros just getting started. But her message is meant to encourage those just starting out to speak up: “QA knows the most about the product because we know the customer’s perspective, and we’re testing the full product from the front end that everybody sees all the way to the backend … the main thing: You want to make sure you have the confidence in your work,” Lee said.

    When quality engineers have confidence in their skills, they’re better able to collaborate with the rest of the software development team to improve testing strategy and triage defects. But building confidence can be challenging, especially as software testing has embraced coding skills to the point where software developers can specialize in quality as SDETs. However, that doesn’t mean that coding experience is the most essential skill in a quality engineering career. Since quality engineering is all about connecting code to user context, soft skills like empathy are just as critical for the next generation. So, even if QE professionals aren’t expert coders, they can build confidence in their skills by understanding their unique place in the software development process.

    Marjie Carmen, software engineering manager at Arch Insurance, agreed. “When I started out, it was commonly believed that software testers didn’t need programming skills or software engineering skills since the focus was on manual testing. But today, I see QA working hand-in-hand with and having almost the same skills as a software engineer; sometimes even more. I really see quality assurance as equal because they have to know the business side and the perspective of the users and where the industry is going. They have to make really good recommendations about how the product should be and need the confidence to go to the engineering side and say, ‘What’s the fastest way I can get this product tested and in the hands of the user?’”

    Creating Unconventional Career Paths in Software Development

    How has hiring for a tester today changed in the last decade? As quality engineering continues to grow as a profession, many in the field are creating new career paths by moving between different roles throughout the software industry. By supporting these unconventional candidates, Maaret Pyhäjärvi, principal test engineer and prominent QA speaker, said that software development organizations can grow internal talent and help quality engineering develop as a field.

    “We hope that there’s more of this openness for moving from one technical role to another,” said Pyhäjärvi. “There’s always the possibility of learning small slices of technicality. Some of my favorite architects these days are so good at software testing; so good at programming and can really bring the whole team together. So we need to allow that kind of movement where people get to try things and break away from stereotypes.”

    Susan Marie, president of the Triangle Software Quality Association (TSQA) said, “I love the capital-Q quality emphasis because it gives me leverage to do well—that’s just one area of quality. I hate the idea that the job title is the one thing that you’re allowed to do; you need to be giving guidance around what’s expected of you and what other people can expect from you.”

    While technology is always evolving, former Slack senior engineering manager Karine Sully said the main characteristics for the job have been consistent—even as far back as a decade ago. “I always look for people who are curious and who have good communication and organizational skills, especially coming in at entry-level or lower-level. All you need is to be able to think critically and organize your thoughts and be able to communicate what it is you’re doing. Then, you can push beyond that and grow.”

    Given the challenges inherent to working with multi-disciplinary teams, Lee said, “I think what helped me the most is having a sponsor. So my advice for anyone on my team, or anyone that’s looking to grow in their career, is making sure you have someone a few levels above you looking out for you.”

    Fostering Diversity in Software Testing Teams

    Growing responsibilities equal a growing number of requirements and skills for quality engineering careers. Navigating the hiring process is never easy, but succeeding in a rapidly changing field can feel like running a marathon at a full sprint, especially when trying to foster more diversity in software development and testing.

    “…Diversity and inclusion are definitely very dear to my heart, and I feel you have to go where people don’t look like you. So if you want to hire more Black or Hispanic engineers, you have to go to areas where you see more of them,” Lee said. “If it’s engineering that you’re looking for, do your research and find those schools that specialize in that particular area. I’m a part of the National Society of Black Engineers, so I tap into that network if I’m trying to help build a diverse team.”

    Carmen agreed that hiring managers need to look closely at areas where there may be holes to fill. “I got into management by looking at gaps and filling them—it was always my goal to find a gap and say, ‘Hey, I can take that off your hands and in the role.’ So it’s a natural evolution to get the different roles that I’ve had by saying yes to more and different responsibilities.”

    Conclusion

    Though the QA and software testing fields are in the midst of several transformations, the change is opening the door for more people to have successful careers in software quality. Though their role is to ultimately improve the product, that road to quality is usually filled with bugs and unpredictable challenges. When quality engineering professionals are confident in their work, it’s easier for them to lead constructive conversations on improving the product—and facing down the engineering teams, if necessary. Building a diverse and confident next generation for quality is essential to building delightful digital experiences for all.

    “I heard from a very famous pioneer in the engineering industry about QA that ‘We don’t have a magic wand; we can’t test quality into your product, but we can test your product.’” Pyhäjärvi said. “The whole concept of QA engineer, QA quality assurance, QE, I think gives people the illusion that we can do something impossible. We can only show light to the areas of the product that need work.”

  • What Is User Acceptance Testing and Why Is it so Important?

    What Is User Acceptance Testing and Why Is it so Important?

    Anyone involved in software development knows how important it is to ensure a positive end-user experience. One way to do that is through the use of user acceptance testing (UAT). 

    UAT is one of the types of testing that you can execute as part of your overall quality assurance process. 

    Source

    It’s essential in helping you determine if your product is market-ready and if it can perform effectively when used by real end-users. 

    UAT helps you avoid implementing software, iterations, upgrades, or customizations that are filled with pesky bugs that can destroy the user experience and render your software unusable (not to mention the damage done to your business’s reputation). 

    When the stakes are so high, it’s clear to see why businesses need a process such as this. In this article, we take a look at user acceptance testing, why it’s so important, and six ways brands can use it to their benefit.

    What is User Acceptance Testing (UAT)?

    User acceptance testing (sometimes known as end-user testing, user acceptability testing or beta testing) is the last step in the entire testing process of any software.

    In product management terminology, acceptance testing takes place as beta testing whereby a small group of early adopters test out the platform and provide user feedback to let you know if the solution meets user expectations and business requirements. 

    It’s done after the system, functionality and all other tests are completed and before the product is launched. 

    The image below shows some of the quality assurance (QA) tests that precede user acceptance testing. 

    Source 

    As you can see from the image, as development work moves forward, UAT progresses from unit testing to integration testing, then system testing and, finally, acceptance testing.

    This final stage is arguably the most crucial element of software development because it ensures a positive end-user experience. 

    As its name implies, it helps you determine if your platform will be accepted by end-users. 

    Testers (end users) work from a checklist of expected outcomes from the test platform and then record what happens when they attempt to execute the different tasks, such as:

    • Logging into the user’s account
    • Placing an item into a shopping cart
    • Checking out 
    • Buying multiple items that qualify for free shipping

    … and so on. 

    If the expected outcomes are achieved when each task is executed, then the software is approved for release. If not, it’s sent back to the developers. 

    This testing process requires collaboration between different stakeholders including project sponsors, business owners, analysts, QA teams and developers. 

    Why is UAT Important to Software Development?

    Although there are many ways projects can benefit from UAT (some of which we’ll look at below), this process is often overlooked by developers. 

    Many see it as a time-consuming and unnecessary process, which is unfortunate because acceptance testing can actually help improve software quality as well as the overall acceptance rate of any newly implemented software. 

    Think about it: You would never drive an untested car, would you? Similarly, businesses should not take risks by implementing software that has not been tested. 

    Considering the time and money you stand to save by ensuring that your software works in accordance with a predetermined test plan, it’s clear that this is something developers need to take more seriously. 

    In fact, far from being the time drain that many developers think it is, UAT can actually gain you a lot of time. How? By preventing unnecessary waste. 

    And far from being a one-off when you come to the end of the project, user acceptance testing extends beyond business application projects because of factors like additional phases, hotfixes, releases/further modifications or customizations and upgrades. 

    Take a look: 

    • Additional Phases: If you choose to roll out at this stage or upgrade an existing application in a phased approach, each phase can have different types of users, different companies (for parent companies, acquisitions, mergers, etc.), additional locations, etc., and you’ll need to plan and conduct separate additional acceptance tests for each individual phase.
    • Hot Fixes: Bug reporting or developer testing during regular day-to-day operations can lead to hotfixes. However, these bug fixes will have to be tested by end-users to make sure that they actually solve the problems originally reported.
    • Further Modifications: Software creators regularly release updates, including limited distribution releases (LTR) and general distribution releases (GDR). LTR releases usually fix issues in a specific problem area, whereas GDR releases are typically heavily tested before release. Both need to be tested to ensure that they fit users’ real-life scenarios so you can achieve your business objectives.
    • Upgrades: Everyone wants to offer their customers the best and most advanced experience. That’s why, for the most part, brands update software regularly to follow industry trends and keep up with the latest innovative technology. Regardless of the upgrade approach you take, whether it’s a little or a lot of customization, it’s crucial that your plan is completed according to business requirements. This will help ensure that the project’s end result is in line with the original business requirements.
    • Privacy Policy Updates: Many end-users may enjoy the overall functionality of the software, but they may desire increased integrations with other platforms or the ability to use a different payment method than that which is currently accepted. While these upgrades and modifications may be easily completed, you may need to alter the information you input into privacy policy generators to ensure that your users legally consent to any new use of their data or access to it by third parties.

    Now that you understand what UAT is and why it’s so important, let’s look at some of the benefits of user acceptance testing to give you a better idea of all the different ways you can leverage it in your own business. 

    Benefits of User Acceptance Testing

    At its core, user acceptance testing will be pivotal in the successful launch and/or maintenance of any type of online platform, whether it’s a simple mobile app or something more complex, such as an eCommerce website. 

    If the test object doesn’t work as desired, the faulty business functions are documented and the item is sent back to developers for repair. 

    Going through this final check ensures that your platform is well-built and performs as expected and that users accept the finished product. 

    Listed below is a summary of the top benefits of UAT: 

    • Benefit: The UAT process allows you to demonstrate that the software’s required functions are operating in a way that’s suited to real-world use and circumstances. As such, acceptance testing focuses on software performance and user behavior to ensure that interactions between the test software’s functionality and the end user’s thought process are aligned. 
    • Benefit: Furthermore, you’ll be getting a much better piece of software because it will be tested and verified by the same people who will be working with it every day. You’ll get optimal opportunities to quickly identify and repair any broken features. So, it’s easy to see how the time you invest in these beta testing cycles will lead to significant improvements in the quality of your software.
    • Benefit: Not only that, but the quality review provided by UAT is also a great way for stakeholders to secure a healthy return on investment (ROI) out of projects. Also, it keeps maintenance costs low and it’s cheaper and faster to fix usability/functionality issues early on.
    • Benefit: You’ll also be able to ensure that your software integrates with third-party platforms including various website builders, email marketing software, project management tools, communication apps and everything else you or your end-users might need to effectively use the software. If your software is supposed to allow for social media sharing, for example, you want to ensure that end-users are able to easily share without going through multiple steps that decrease an otherwise positive experience with the software.

    As you can see, user acceptance testing is vital to the successful deployment of new business applications and software enhancements, as well as to ensure organizational efficiency and success.

    The process helps you to mitigate risk and boost the ROI of your project. But most importantly, UAT ensures that your customers don’t end up being acceptance testers by default, which would have a negative impact effect on your business.

    6 Ways to Use UAT

    Here are six ways you can use this process to achieve better outcomes.

    1. Use user acceptance testing for internal development.
    2. Use the testing process to review new features.
    3. Invite users to take part in the tests to help them become more invested in the brand and become brand ambassadors.
    4. Use beta tests as a premium feature to convert subscribers to higher tier packages, increasing sales for the company.
    5. Share information about UAT and its results on social media platforms, email and other channels as part of your digital marketing campaigns.

    These are just a few of the different ways your organization can benefit from UAT far beyond the initial goal of the testing process. 

    And remember, it’s all about the users. 

    UAT isn’t just another testing technique. This test puts the user at the center of the discussion to answer questions such as:

    • Are users able to use the software without error(s)?
    • Does the solution meet promised specs? Does it behave in an expected way?
    • Does the software continue to work when users take unexpected actions?

    …and so on. 

    Whether the platform is constructed in-house or by a third-party company, all solutions must go through user acceptance testing because your users are the ultimate audience who will pay for your solutions. For this reason, it’s worth taking the time to ensure that your software solution meets both user expectations and business requirements.

    Planning and Managing the UAT Process

    The key to a successful UAT testing process is to understand the needs of your target users and their expected behavior. This way, you’ll be able to develop defined acceptance criteria so you can come up with efficient tests to complete the testing process fast. 

    Also, since the goal is to assess the market readiness of the product, it’s important to record all evidence and track progress with the dev team as well as report to stakeholders so everyone is on the same page. 

    With that said, here are five steps to help you plan and manage the user testing process: 

    Step One: Planning. The user acceptance testing strategy mainly includes a time frame outlined during the planning step.

    Step Two: Designing Test Cases. Test cases cover the core functional frameworks of the platform in real-world usage. When designing test cases, you need to do the following: 

    • Determine the needs of target users and stakeholders 
    • Design test cases according to the user journey and product specifications 
    • Summarize test runbook, acceptance criteria, etc. 
    • Prepare test data, e.g. registration, three days in use, one week in use, etc. 

    Step Three: Setting Up Your UAT Team. The acceptance testing team is comprised of real-world end-users or users who are not part of the development or testing phases to ensure objectivity. You should never use developers to test their own software as they may bring bias (albeit unintentionally) into the testing process. 

    Step Four: Executing and Documenting Test Cases. The UAT team executes the test cases, together with relevant random exploratory tests, after which all bugs are logged into the bug tracker along with steps to reproduce.

    Step Five: Bug Fixing. Once you’ve executed the test cases, you need to keep track of the tickets and user feedback. The development team can account for all the bugs found by the testers and start fixing bugs depending on severity. 

    Step Six: Confirm Fixes and Get Stakeholders to Sign Off. Once all UAT tests have been run and the results are in hand, the acceptance decision can be made. If all users are satisfied, then it’s a go. If not, it’s a no-go. 

    Last But Not Least

    UAT is the last stage in the software development life cycle. But, it’s also one of the most important things you can do to ensure the success of your software. Just keep in mind that for this process to be effective, it should be viewed as validation and not verification. Use your test procedures to validate functionality in both smaller scope (as a way to test functionality) and in a larger context (to simulate real-world processes) so you can get an end-user vision of your product which will ultimately help you increase the robustness and usability of your software.

    Are you ready to use UAT to get feedback on your software so you can increase end-user happiness? Share your thoughts below!

  • ShiftLeft Report Reveals State of Application Security

    ShiftLeft Report Reveals State of Application Security

    A report published today by automated application security testing platform ShiftLeft found only one in three applications has an attackable vulnerability. The report also found organizations that prioritized their remediation efforts based on the level of actual threat are fixing 76% of those vulnerabilities within two sprints lasting 12 days, on average.

    Based on millions of scans performed in the last year using the ShiftLeft platform to identify vulnerabilities, organizations have reduced the number of vulnerabilities they need to address by 97% on a year-over-year basis, the report found.

    In addition, the report noted that the average median scan time is only 90 seconds, which helped drive an overall 68% increase in daily scans year-over-year. Overall mean-time-to-remediate (MTTR) has dropped by 37%, the report also found.

    ShiftLeft CEO Manish Gupta said given the increased number of vulnerabilities, it’s more critical than ever for organizations to focus their remediation efforts on those that can be exploited within their IT environment. For example, the report found that only 4% of all Log4j instances discovered in the IT environments analyzed were vulnerable. Rather than spending weeks looking for every instance, Gupta said IT teams would be much better off focusing their efforts on the instances that might actually be exploited.

    The challenge is that not enough organizations have a way to score vulnerabilities based on the risk they represent to the business, Gupta added.

    While there is a lot more focus on adopting DevSecOps best practices in the wake of a series of high-profile breaches of software supply chains, the ShiftLeft report made it clear that presenting developers with a long list of potential vulnerabilities is counterproductive. The bulk of vulnerabilities that are being discovered are not actually attackable, noted Gupta.

    That doesn’t mean application security doesn’t need to improve, but it is clear that a lot of time and effort is being wasted, added Gupta.

    In theory, at least, as the cultural divide been application development teams and cybersecurity teams continues to narrow, the overall ability to prioritize remediation efforts should improve. The first step toward achieving that goal is making it easier to identify what percentage of the application attack surface is really vulnerable. Otherwise, developers presented with a long list of theoretical vulnerabilities will continue to prioritize developing new features and capabilities versus remediating vulnerabilities.

    One way or another, vulnerability remediation issues will soon come to a head. In the wake of an executive order from the Biden administration to improve overall application security within federal IT environments, many organizations will adopt similar policies and processes. Developers, naturally, will inevitably either push back or do an end-run around processes they deem too onerous. Rather than allowing a stalemate to occur, Gupta said savvy DevOps teams will invest more time in identifying the critical vulnerabilities that need to be addressed before cybercriminals exploit them.

    The issue, of course, is that cybercriminals are getting more adept at discovering and exploiting those very same vulnerabilities.

  • Codenotary Adds SLSA Framework Support to Advance App Security

    Codenotary Adds SLSA Framework Support to Advance App Security

    Codenotary this week announced it has integrated support for the Supply-Chain Levels for Software Artifacts (SLSA) framework in its free notarization and verification service for ensuring the integrity of code.

    Moshe Bar, Code Notary CEO, said as the first application security platform to attain SLSA compliance, the company is making it easier for organizations to secure their software supply chains.

    SLSA is a framework defined by Google that it uses to attach binary authorization to code. In its current form, SLSA is a set of guidelines that identify how to attain four different levels of application security. The long-term goal is to create a framework that will automatically generate auditable metadata that can be fed into policy engines to create a SLSA certification.

    Codenotary provides tools for cataloging and verifying the authenticity of components used with a software development life cycle. At the core of that platform is an immutable open source immudb database that cryptographically attaches an identity to each software artifact. That capability can also be used to dynamically generate a software bill of materials (SBOM).

    The Codenotary Attestation Service can also be integrated with the continuous integration/continuous delivery (CI/CD) platform that DevOps teams use to build and deploy applications using those artifacts. Ideally, DevOps teams should construct a security pipeline that runs alongside their existing DevOps pipelines, noted Bar.

    In the wake of recent high-profile software supply chain breaches, the level of scrutiny being applied to how software is constructed has increased significantly. In theory, at least, developers have assumed more responsibility for application security as part of an overall shift left that provided them with more programmatic control over IT environments. Cybercriminals, however, are now becoming more adept at compromising developers’ credentials and insert malware as an app is developed.

    DevSecOps best practices are, of course, being adopted to teach developers how to better secure their application environments. The issue is that not enough developers are learning how to secure applications fast enough. The volume of attacks against the software supply chain continues to grow at a rate that is now much faster than organizations can thwart simply by educating developers.

    There is a now clear need to embed more cybersecurity guardrails within DevOps workflows, noted Bar. Rather than attempting to achieve that goal in a piecemeal fashion, Bar said Codenotary is making a case for an integrated platform that can be invoked as a natural extension of a DevOps workflow.

    In general, Bar said there’s now a greater sense of urgency when it comes to software supply chain security. Many of the reviews of those processes are, arguably, long overdue. One way or another, there will be a lot more questions asked about how software is actually developed. The organizations that consume this software are starting to appreciate how vulnerable they are as the overall dependency on software continues to increase. The only real question now, unfortunately, is how many more breaches there might be before those software security issues are thoroughly addressed.

  • Threat Modeling as a DevSecOps Practice

    Threat Modeling as a DevSecOps Practice

    Software engineers are always under pressure to build more software, faster.  At the same time, there is increasing regulatory and market pressure for secure software that meets users’ and regulators’ requirements for data privacy.  This dynamic often puts software engineers at odds with application security or product security teams.

    In fact, 81% of developer teams at large enterprises have launched applications that feature insecure code, according to one researcher, with half doing this on a regular basis. The firm argued that developers and security architects need a better relationship, and I couldn’t agree more.

    These technical teams are working towards the same goal, so it’s in everyone’s interests—particularly senior leadership—to create a more cohesive working environment by breaking down barriers between teams. If the security of a system is designed and enforced by an external security team—disconnected from the rest of the engineering team—then this only reinforces security as a separate silo. But companies can ease tensions by embedding the responsibility for security in the engineering teams themselves: This is the thinking behind DevSecOps. Thinking about security starts when engineers think about solving the problem: At design time.

    While the shift left movement was born two decades ago with the goal of ensuring product testing happened early in the development stage, I believe we need to start left to make real change. In adopting this culture, security and privacy will be less of a bolt-on afterthought and a built-in fixture within the development process, ensuring that engineers and security teams alike are aligned and have visibility of each other’s objectives. 

    By reimagining the build, test, fix, build, test, fix cycle through building an architecturally secure design in real-time, technical teams can detect software vulnerabilities sooner rather than later, which means more efficient product deployment. This is where the process of threat modeling as a business tool can be of great benefit.

    Historically, threat modeling was a static, slow and manual process conducted on whiteboards, as part of a workshop led by a third-party security expert who lacked institutional knowledge of the inner workings of the company. However, this has evolved dramatically into an easily implemented, automated security practice that can be baked into the development and security cycle from the outset, empowering organizations to start left.

    Through automated threat modeling, the security workload burden is relieved for both security architects and engineers. Companies are, in turn, better positioned to keep pace with the cadence of ongoing software rollouts, as an automated platform suggests security improvements as an application evolves. The launch phase bottleneck that’s typically created during the testing stage is therefore removed, as security requirements are identified prior to development—which means that risk is managed before a line of code is even written. Based on this, it should quickly become apparent what a time- and cost-saving exercise threat modeling is.

    No developer or security specialist wants a flawed product to hit the market, so it makes good business sense to adopt a new approach that will lead to better outcomes. Enterprises that introduce threat modeling as part of their development and cybersecurity processes will be on course to embed a true DevSecOps system within the company, transforming the relationship between these two departments.

    In my experience, culture and growth are the key factors for enterprises that decide to adjust their cybersecurity diet to include automated threat modeling. Convincing the app dev teams to invest time into the process is, by far, the most significant cultural hurdle to overcome, because they need to witness the benefits—and up until this point they’re disconnected from the AppSec process.

    Once equipped with the tools and knowledge, delivering secure design will be second nature to developers and a new collaborative culture will be created. In essence, the more teams are given oversight of and accountability for software design flaws, the easier it is to recognize how vital these preliminary checks are.

    Likewise, there comes a stage of business growth when the ‘security first’ and ‘start left’ mindsets are essential to keep enterprise project management on track—and this ties into the culture. As the rate of scale within a company begins to increase and the pace of software deployment must quicken in line with customer demand and market competitiveness, discovering flaws at the end of the design process just isn’t an option. 

    With developers and security teams responsible for vast amounts of code, product resilience on delivery is crucial and ensuring a robust DevSecOps culture is the route to success.