devopsy.kinsta.cloud

Tag: security testing

  • The IT-DevOps Life Cycle is Like a Pyramid That Keeps Growing

    The IT-DevOps Life Cycle is Like a Pyramid That Keeps Growing

    Don’t let anyone tell you how much easier IT life is these days. It is, in many ways, more fun and productive … but it is not easier. In fact, we’re doing more with less, cubed, at this point. My point is—think of the life cycle as a pyramid. The base is the build++ chain, from VCS to public-facing active protection; the height is the new technologies and systems being bolted on daily—like SaaS tools someone in accounting just signed up for or the new architectures a dev team in a subsidiary just decided to adopt (but which will eventually have to be maintained by IT). And the depth is the completely new capabilities that are required by market forces or regulation – SBOM is the poster child for this group; suddenly, just about every vendor is giving it to you as part of their product.

    No matter which of these axes is growing, the workload is generally growing commensurately. Think about it: If a new product is implemented, it impacts IT’s workload. If a new technology that solves a problem is implemented, it, too, impacts workload. And if a new capability in an existing product is dictated by the compliance team for any reason, this, too, increases workload. This is just the business side, and doesn’t include the fact that most orgs are dropping newly developed applications on a pretty regular basis these days.

    Automation has helped, but it hasn’t solved the problem yet. I believe it can, given time and focus, but for the most part, the demand for new technology to solve age-old problems (“We can’t effectively test” being met with “Here is automated test generation, execution and results filtering”) has been a net increase in workload. Though far smaller than it might have been, it is still increasing. This is true across IT. The test example above applies to app dev and security, with result sets that did not exist six years ago because the process was too expensive in staff hours. It also applies to ops because the processing power to get those tests done – and it is a lot with large test environments – is ops’ responsibility.

    Painting a bleak picture? Nope. Making certain we’re realistic. Trying to ensure that we consider how much we are already doing, and take a chance to implement actual time savers. For example, applying that same test automation to testing (for most orgs, I’m thinking security testing) that was actually being done already will reduce the cost of getting it done. Using SBoM for SCA to contribute to existing compliance saves a lot of time generating that information. Using new coding assistants shortens coding time and (arguably at this point, but inevitably, given the fullness of time) improves code quality, reducing rework, which is costly, even in an agile environment.

    We are growing our capabilities almost daily. I’m just suggesting that, in the process, we backward-apply the good stuff to reduce existing overhead. And as always, this big, complex beast exists and keeps clipping along because you’re there, keeping it fed and stable. Keep rocking it, and save yourself some time by using new technology smartly—there is plenty more to do than whatever you streamline.

    Image source: Photo by Michael Dziedzic on Unsplash

  • Synopsys Preps Extensions to Polaris SaaS Platform

    Synopsys Preps Extensions to Polaris SaaS Platform

    Synopsys plans to extend the capabilities of its Polaris Software Integrity Platform for securing application development environments by adding dynamic application security testing (DAST) tools along with the ability to scan code used to provision infrastructure.

    As a software-as-a-service (SaaS) platform, the Polaris Software Integrity Platform was created by combining the static application security testing (SAST) tool Synopsys gained with the acquisition of Coverity in 2014 and the software composition analysis (SCA) tool it added to its portfolio by acquiring Black Duck Software in 2017.

    In 2015, Synopsys acquired the Seeker tool for analyzing the code created using interactive application security testing (IAST) tools from Quotium. The company last year acquired WhiteHat Security from NTT to add a DAST tool.

    Patrick Carey, senior director for marketing strategy for Synopsys, said that the company would add the capabilities of those latter two platforms to create a comprehensive set of best-in-class security tools that are easily accessible via a SaaS platform.

    At a time when more organizations than ever are moving to secure software supply chains, Carey said it’s critical to provide developers with more visibility into vulnerabilities across the entire software development life cycle (SDLC). Some organizations are opting to achieve that goal by embedding tools within the SDLC themselves, but many organizations would likely prefer to invoke a SaaS platform that they don’t have to maintain, noted Carey.

    He noted that a SaaS platform makes it simpler to point a collection of integrated tools at a code repository to surface those vulnerabilities.

    As DevSecOps workflows continue to evolve and mature, more responsibility for application security is shifted left toward developers. The challenge is that many developers have limited cybersecurity expertise, so it’s critical that they have access to high-quality security tools.

    In addition, a SaaS platform helps reduce the total cost of achieving that goal by providing multiple capabilities within a single integrated platform, said Carey.

    Naturally, most of the focus today is on reducing the number of vulnerabilities that find their way into production environments as applications are being built. However, there are already large numbers of unaddressed vulnerabilities in applications that have already been deployed. Organizations will need to find ways to address that technical debt in addition to providing developers with tools that help prevent those code vulnerabilities in the first place.

    As such, there is no one central place through which application security can be managed, noted Carey. Instead, application security needs to be addressed across every phase of the SDLC, he added.

    It’s not clear how quickly organizations are adopting DevSecOps best practices, but as calls for increased liability for software development increase—as outlined, for example, in the U.S. National Cybersecurity Strategy— organizations should expect DevSecOps workflows to become a requirement. Otherwise, organizations will be severely penalized for vulnerabilities that find their way into code. In effect, there is now a race to embrace DevSecOps before any proposed liability legislation becomes the law of the land.

  • At Some Point, We’ve Shifted Too Far Left

    At Some Point, We’ve Shifted Too Far Left

    Those of us involved in DevOps have a tendency to see the world with blinders on. It is rather easy to fall into the “If all you have is a hammer, everything looks like a nail” trap. We have used the phrase “shift left” with the attitude that this is the solution to every problem in IT.

    Just for clarity, it is not. That is as blunt and unambiguous a statement as I can make. DevOps is great and has helped in many areas, but it is not the only solution, and not every problem is a DevOps toolchain problem.

    In the end, much critical work in IT is not DevOps-related, nor can it be impacted by DevOps. I’ll mention two areas that are obvious and talk in-depth about one of them.

    First off, no matter how far down the Agile/DevOps trail you travel, the wagon you are riding in is separate and distinct from the DevOps process. If you have a data center, all that hardware is still, in the end, hardware that must be configured and managed. For the “But we’re 100% in the cloud!” crowd (who say that loudly and forcefully enough that I suspect they’re trying to convince themselves that this is a good thing)—in the end, you are still using a network and computers to connect. The massive work-from-home migration due to lockdowns made it abundantly clear that IT is needed to set up and support both. Some of the horror stories coming across ops and DevOps social media were … ugly. Employees don’t want to be—and should not need to be—IT specialists to get their job done.

    The other easy one is security. Yes, insert security into development and DevOps workflows. Yes, make operations staff more aware of security. Yes, automate security as much as possible because we’re chronically short of security staff. But, in the end, lots of security is external to DevOps. While Kubernetes security is absolutely DevOps, password management, for example, absolutely is not. And that’s just one of many, many examples.

    So I am going to ask (but not answer) the question: Where is the line? How far left is too far? The answer is yours to decide with regard to your organization and the tools/environment that exist in your organization. In a primarily cloud environment, the answer to this question is far different than in a primarily data center environment. It is rare that any organization be purely one or the other at this point in time, so there are similarities, but few exact solutions we can name.

    Security is akin to a cost center. It is actually insurance, but few orgs view it that way. And even if they did, it would not make much difference. Much as my father once declared that he and Mom were “insurance poor” and canceled half their policies, most orgs want to contain the costs of security whether they admit it or not. When ‘security merged into DevOps’ hits the ears of people trying to stretch budgets just as far as they can, bad things tend to happen.

    DevSecOps implements security where it should have been all along—at the development stage—but it does not replace other security, it just makes our apps more protected. There are cases where security testing hours are seriously reduced, but since testing in all forms is the first thing to go when staffing is tight, again, it is not replacing something that was happening—it is making it possible for us to do what we should have been doing all along.

    It doesn’t help that our toolchains have taken on lives of their own—at many organizations, staff are required to maintain them—because they, too, are apps that DevOps struggles to help. Once you’ve invested time and money in building that edifice, you want to use it as much as possible. So, many orgs want to shift left everything in IT, right into the toolchain octopus.

    You know your org better than anyone. You are the ones making it work day in and day out, in all of your environment’s glorious complexity. Start talking about what is too much; when adding a step in an already complex toolchain needs to be justified. Push back against everything needing to shift left (where appropriate, I’ll say again for clarity, DevOps has done amazing things for us, but too much of a good thing is our living nightmare and, historically, is tomorrow’s technical debt). Keep rocking it, and make sure you are rocking the right things in the right places.

     

  • Where Has All the Testing Gone?

    Where Has All the Testing Gone?

    Testing has long been a problem child of IT in general, AppDev in particular, and now it is DevOps’ problem. There are things that DevOps can do to improve the chances of tests actually occurring for your application(s). Interestingly, there are some that think we no longer need testing; the argument is that because new releases can be cranked out so quickly if something needs to be fixed that it is irrelevant. Combining feature flags with the ability to roll out a new version and, well, there may be some truth to that—at least for a limited subset of testing.

    If an organization implemented feature flags for user acceptance and new feature testing and then standardized on test-driven development for DevOps development teams, that would leave systems and security testing. Systems testing could, theoretically, be replaced by the ability to roll out a new version. Although systemic issues can be complex and difficult to fix, this is a riskier proposition than extending test-driven development to the systems level. For the sprint, assign a person responsible for the overall integrated system and have the rest of the team support them through the process of making certain all works as expected.

    Testing—particularly systems testing—slows releases. I get that. The rate at which most of us can iterate once DevOps systems and processes are in place is fast enough that you can afford a bit of slowdown. This leaves security testing as the last standalone bit. But some security validation can happen during the development process if DevSecOps tools are being utilized. That still leaves some security issues to test for but does remove a large chunk.

    The key to these last bits is to get them integrated into the process/toolchain. DevOps runs straight through, over and over. Any bit worked into the “runs straight through” will be executed. If today’s iteration falls for the “We ran over, we’ll make the deadline with less testing” mentality; not an issue—it’s one iteration. Just don’t let that become the default answer to someone not completing an iteration’s work on time. Development can have unexpected speed bumps that delay delivery and acknowledging that and documenting what it was for future reference is important. Covering it up by reducing testing time risks a less stable product and does not help overcome the issue in the future.

    So start thinking in terms of DevOpsTest. Make sure the product you are delivering represents the massive time investments you’ve made producing and maintaining it. You’re rocking it—and something like integrated testing ensures that users see how great you are rocking it and not an error page.

  • Continuous Testing Practices – Part 3

    Continuous Testing Practices – Part 3

    In my prior blog, Continuous Testing – The Quest for Quality at Speed, I described five tenets and some of the practices for continuous testing to help with understanding what continuous testing is. In my consulting work, I find it necessary to use 15 categories of practices to assess an organizations’ continuous testing capabilities. Given the large number of practices, I am publishing the complete list in three parts.

    This article is part three, which covers the following categories of continuous testing practices.

    • Security tests
    • Pre-flight testing
    • Integration testing
    • Pre-production testing
    • Test in production

    You can read part one here and part two here.

    Security Tests

    • Credentials and secrets needed to operate any test are vaulted.
    • Customer sensitive data in application databases used for testing is obfuscated from misuse by people involved in testing.
    • New security tests that are necessary to test a software change are created together with the code and integrated into the trunk branch at the same time the code is. The new tests are then used to test the code after integration.
    • Security tests for each DevOps pipeline stage are automated and may be selected automatically according to predefined criteria.
    • Test results that indicate possible security concerns are tagged for security analysis.
    • Dynamic or interactive application security tests exercise the application to check for security vulnerabilities. Results of these tests are delivered to developers through tools and feedback loops native to their organization.
    • Attack patterns, abuse cases and tests are built according to application module profiles.
    • Unit, functional and integration tests around—and especially outside—boundary conditions are tested. Tests include error handling, exception handling logic and negative tests.

    Pre-Flight Testing

    • Products are architected to support modular independent packaging, testing and releases. In other words, the product itself is partitioned into modules with minimal dependencies between modules. In this way the modules can be built, tested and released without requiring the entire product to be built, tested and released all at once.
    • Prior to committing software changes into the integration or trunk branch, pre-flight tests are conducted on the software change integrated with the tip of the integration branch within a private instance, to ensure the changes will not break the integration branch.
    • Requirements for testing are included in agile backlog planning procedures.
    • Pre-flight tests include unit testing, static code analysis scanning, functional, regression and performance tests. During this stage, feature flags or toggles may be installed in the software and each flag setting is verified to make sure the flags work. New automated tests that validate the changed software are created and tested, for use in later stages of the pipeline and subsequent regression tests.
    • Pre-flight tests, conducted prior to integration, include unit testing, static code analysis scanning, functional, regression, performance and security tests. During this stage, feature flags or toggles may be installed in the software and each flag setting is verified to make sure the flags work. New automated tests that validate the changed software are created and tested for use in later stages of the pipeline and subsequent regression tests.
    • Prior to committing software changes into the integration or trunk branch, pre-flight tests are conducted on the software change integrated with the tip of the integration branch within a private instance to ensure the changes will not break the integration branch.
    • Test-driven development (TDD) is used to ensure tests are created ahead of the code to be tested. TDD is most often applied for unit level tests but may also be applied for functional testing. A key advantage of TDD is that it ensures working tests are available for use in all pipeline stages. Without using TDD, there is a tendency for test creation to fall behind the code. Another advantage of TDD is that tests are created without the bias of knowing how the code was written, because it hasn’t been written yet.
    • From a testing point of view, microservices introduce the requirement to validate the contract of each service with any other services that use it. The independence of the microservice or any interservice dependencies need to be well tested. Performance and reliability considerations of operating services over a network need to be verified. If the application changes, affected microservices, or any dependent group(s) of microservices also need to be regression tested.
    • Containers offer the opportunity to package test resources in special test containers so they can be conveniently and immutably invoked on-demand for testing changes with elastic scaling benefits.
    • A strategy ensures that changes to a database, or changes to an application that uses a database, perform as expected, during each stage in the continuous delivery pipeline.

    Integration Testing

    • Software changes are tested incrementally by a cross-functional team over all stages of the continuous delivery pipeline.
    • During the code commit stage, assessments are conducted on the committed code to validate that the pre-flight test results satisfy acceptance criteria for integration into the integration branch.
    • During the build stage, a select group of tests are run on each build to ensure the integrated build satisfies build integration acceptance testing criterion.
    • Containers offer the opportunity to package test resources in special test containers so they can be conveniently and immutably invoked on-demand for testing changes with elastic scaling benefits.
    • API contract testing is used to test microservices interfaces.
    • Test code and data for each application are stored and maintained in test containers so they can be managed using the same container creation, orchestration and monitoring capabilities as the application.

    Pre-Production Testing

    • After the integration and build stage, functional, performance, regression, customer acceptance and security tests are conducted to ensure the images resulting from the build or set of builds satisfy functional, performance, security and delivery assessment criteria.
    • Testing during deployment include dark launch strategies such as A/B testing. feature flag rollouts, green/blue deployments and canary testing.
    • Release assessment criteria are defined and use test results to decide whether a release meets release acceptance criterion.

    Test In Production

    • Testing during deployment uses strategies that minimize the impact of failures such as A/B testing. feature flag rollouts, green/blue deployments and canary testing.
    • Automated tests verify each deployment.
    • Chaos engineering is used for testing the resiliency of applications, deployment infrastructure and pipelines.
    • Tests for disaster recovery and restoration of services, infrastructure and pipelines are conducted periodically.
    • Fire drills on applications, pipelines and infrastructures are used to test procedures for detection and recovery of applications, pipelines and infrastructures.

    What This Means

    Continuous testing (CT) practices are critical to success with DevOps. Many organizations underestimate both the importance and the practices needed to engineer Continuous Testing. This article is part of a three-part series that provides a comprehensive list of practices that can be used to understand and assess an organization’s Continuous Testing capabilities. To learn more about my blueprint for continuous testing, refer to my book Engineering DevOps.

  • Make a Plan for Test Automation

    Make a Plan for Test Automation

    The amount of testing that we could be doing is massive. Most of us don’t look at testing across the spectrum and all-inclusively, but let’s do that for a second.

    We have functional testing at the code level, which is reasonably well automated already, if your shop is using such automation. Then we have integration testing, which is a superset collection of API testing, code integration testing, usability and regression testing. Integration testing is automated, and sometimes that’s done well, but this massive space is still growing. Next up would be security testing, which covers much of the same code that other testing is interested in from a strictly security focused perspective. Then, there is actual usability testing, which is more important for some areas than others—embedded needs far less usability testing than web apps, for example—and has different levels of automation at this point in time.

    Really, when you think about it, the complexity of this environment is massive. We not only need to test an array of different functionalities from an array of different perspectives, but we have to do it on an array of platforms. Testing on Windows is different than testing on Linux; testing in the cloud can use different tools than testing in-house; clients can be on any given platform, depending on the application and target market … we really do have a beast, here.

    And testers have always been behind, or have not covered things well. This is not a ding on testers, this is the result of the complexity at hand and the fact that, in many instances, one line of code can go wrong in a bunch of different ways.

    Enter automation, which is growing rapidly in the world of continuous integration (CI). Automated testing tools range from broad to very focused, and all help facilitate testing. The DevOps increase in development means more testing, and testing was already behind. This has been discussed pretty thoroughly in the past, but it finally feels like we’re getting to a point where we have viable solutions to the problem. If testers become test developers that review results of automated tests, and through CI those automated tests are increasingly worked into the process of development, we’ll see testers getting better coverage and quality seeing a continued increase.

    But that does mean you should keep your eyes on the market as it continues to grow, and pick tools that offer the most functionality that is most useful to your test team. And get them up and running. More testing means more work setting up tests. Even no-code test solutions for UI require someone to record the actions that are necessary to complete the test, and that takes someone’s time. But the return is long-term testing that doesn’t require a person until a red flag is raised by automation.

    I’m all for it. Both testing and security have the most to gain from automation, because they’ve been the most behind historically, and corner-cutting has cost organizations dearly. So, let’s get with it; check out some new test tools (there are plenty listware articles giving you a place to start), and let’s make great software better!

  • Prevent False Positives From Derailing Shift Left

    Prevent False Positives From Derailing Shift Left

    Static application security testing (SAST) tools are designed to balance false positives (incorrect warnings) with false negatives (missed vulnerabilities) primarily because deeper analysis requires more time and computing resources. Both of these are in short supply among developers that are tasked with meeting shorter and shorter product delivery milestones.

    So, while SAST vendors consider a true positive a correct detection of a real defect, and a false positive an alert on a bug that does not exist, this is rarely the lens used by developers. What really matters to them is whether the output of a SAST tool is useful and actionable. For example, there is a great deal of variation in how developers interpret SAST scan results, depending on the nature of the defect, the role of the user, the platform on which the application will run and the environment in which it is deployed.

    Take, for example, a true positive result of a buffer overrun, one of the most notorious classes of C/C++ defects from a security perspective. In the early stages of application development, it almost always makes sense to change the code to fix such a bug. Developers are actively changing the code, anyway, so fixing it involves little extra overhead. However, if the same defect is found after the application has been deployed, then it is much trickier to decide whether it is worth fixing.

    That’s because the cost of fixing a real bug may exceed the benefit of fixing it, and the benefit of “correcting” a false positive may exceed the cost of leaving it alone. Since SAST tools can only be relied on to give narrow technical answers, humans must interpret and determine which static analysis results should be acted upon.

    Consider a hypothetical comparison of different SAST tools in the graph below. Tool A has good recall (i.e., the ability to identify real defects) and precision (i.e., the ability to exclude false positives) which results in finding many of the real bugs with a reasonable amount of false positives. Tool B has high precision but poor recall, resulting in low false positives but a higher number of false negatives (undetected security vulnerabilities). Tool C has poor precision but high recall, resulting in detecting all the possible errors but also with a very high number of false positives.

    A comparison of recall and precision of hypothetical SAST tools.

    Developers hate false positives because they introduce unnecessary work and delays. This, in turn, has a disproportionate effect on the way tools are designed, configured and used. If given a choice between Tool A that reports 40 real defects and 10 false positives, and Tool B that reports 50 real bugs but with 50 false positives, users will almost always prefer the former, even though it is finding fewer real defects. This is perfectly understandable — users are asked to weigh an immediate concrete negative (time spent looking at false positives) against an intangible potential future positive (vulnerabilities that may not show up).

    However, if one offsets the time and risk saved in finding those 10 bugs earlier (i.e. by avoiding expensive and potentially dangerous bugs in finished products) against the time needed to assess the additional 40 warnings as false positives, then it quickly becomes apparent that configuration B is more economical. This is especially true with security vulnerabilities that slip through testing and can have expensive consequences.

    Since the number of warnings within a DevOps pipeline can be a deterrent to getting the most out of a SAST tool – especially early in the project, when it provides the greatest benefits – organizations should consider the following techniques to increase developer attention to alerts:

    • Filter and Focus: Parse the viewed data from the tool’s interface, focusing on what is most important for the project and assigning developers to fix critical issues in priority order.
    • Mark and Defer: Lower the priority, or change the state to “later,” for example, on all or a subset of the warnings based on some set of conditions that are less crucial to the project.
    • Stop the Bleeding: Use the above techniques to temporarily defer existing warnings with the emphasis on fixing new defects that are introduced as code is changed or new code is developed.

    Despite the fact that it will invariably produce some false positives, SAST helps shift security further to the left by ensuring security and programming vulnerabilities are eliminated when code is being written, rather than at the testing stage when they are much more expensive to fix. Filtering and prioritizing warnings are two effective techniques to ensure developers don’t throw out the baby with the bathwater.

  • Integrating Security in the Development Process With DevSecOps

    Integrating Security in the Development Process With DevSecOps

    Occasionally, it’s worthwhile to reflect on how we develop and deliver software during times of rapid change and significant disruption. In those moments of reflection, we learn from the exciting trends and changes that are taking place. Recently I had the opportunity to commiserate with two technical thought leaders at Perforce: Stuart Foster, product manager, and Steve Howard, static code analysis evangelist.

    The disruption caused by COVID-19 goes much deeper than just the rapid shift to work-from-home. Development teams are being asked to keep up developer velocity, while the organization as a whole accelerates digital transformation projects, shifting priorities and emphasizing investments needed to successfully navigate difficult and uncertain economic and market conditions.

    In our conversation, we note how investments already made in shifting to the cloud, remote and cloud-based development, automated testing and pipeline automation and security tools are helping organizations today. We also explore how current conditions influence product roadmaps, causing software teams to take a holistic look at their development processes and tools, and organizations to rethink how to take their software delivery capability to the next level.

    Stuart and Steve share their software best practices on collaboration, automation, security, standardization and more.

    Explore our conversation in more depth by viewing the video and transcript below.

    Transcript

    Mitch Ashley: Well, I’m happy to be joined by Stuart Foster, who’s product manager with Perforce, and Steve Howard, static code analysis evangelist, and I think a lot of other things, too—does a lot with Perforce. So, welcome, gentlemen—good to be talking to you today.

    Stuart Foster: Thanks, Mitch.

    Steve Howard: Hello.

    Ashley: You know, we’re talking about—you know, Perforce does a lot of things, and I think we’re just kind of focusing on what maybe some of the current conditions, current best practices are. There’s a lot happening right now in everybody’s world—a lot of change, a lot of disruption.

    But before we get to that, could I have you both introduce yourselves? Maybe Stuart, would you start first?

    Foster: Yeah. Hi, everybody, I’m Stuart Foster, I’m Perforce’s product manager for our static code analysis products.

    Ashley: Great—Steve?

    Howard: Yeah, and I’m—well, I have, as you say, many jobs and roles within the organization, but primarily focusing on the evangelism around using static code analysis for both of our products, Klocwork and QAC, here at Perforce, yeah.

    Ashley: If you’re working with developers, you gotta be doing evangelism, right?

    Howard: Yeah. [Laughter] 

    Ashley: They expect you to give them support, and be talking to them, in a different kind of way. Tell us—would one of you tell us a little bit about Perforce and what all you all do?

    Howard: Yeah, sure—well, I’ll take that. So, Perforce is an organization, obviously as, you know, originally comes out of the version control software. But really, it’s about developer tools and how we can help make the developer’s life easier in the day to day tasks. So, whether that be trying to prove compliance to standards or just trying to get the stuff out the door in the first place—anything that we can do to help make that process smooth and reliable is kind of where we are. And, you know, these days, that has a bit of a—a buzzword, if you will, around the DevOps tooling ideas, but essentially, it’s developer tools and making their life more comfortable.

    Ashley: That’s great, because you all fit into a number of places in the sort of tool chain, pipeline, workflow process across the development life cycle.

    Howard: That’s right, yeah.

    Ashley: So, that’s awesome. Well, let’s get into this. I’m really curious about what you’re finding in market, and one of the things we like to do when I’m talking with experts like yourself is, let me just try to share some of the things that we’ve experienced within the past, but also kinda recently that it’s happening, because we’re all dealing with economic change, disruption, COVID, work from home, all of those kind of things, but you know, we’ve done some research at my company and there’s a lot—of course, a great, great emphasis on acceleration of digital strategies and, of course, that puts a lot of pressure and emphasis and people are betting money on software being the answer to helping them implement their strategies.

    Are you seeing the same kind of thing, that acceleration of digital transformation and software—elevation of software projects?

    Foster: Yeah. We’re seeing that across the board, even in our own company here. You know, a lot of us have transitioned to work from home, and then you have to start thinking about the considerations for that, right? 

    And what we’re finding ourselves and working with our customers is, a lot of people are shifting more technological tools, right, to be able to better produce workflows and code reviews, how to—working from home, security, right? How secure are your remote development work stations? How is security being built into your software and your workflow? So, tools are being used across the board for that. Just being able to rely on a piece of software versus in-office workflow has become more and more important. And amongst all that, then there’s the goal to, you know, keep development velocity up, which could be a challenge when working from home, right? So, we’re seeing a lot more DevOps automation of, you know , a lot of workflow process and things like that.

    Ashley: Yeah, there’s no more popping over the edge of the cube or to the table next to you, right? Somebody sitting there at the café, in the office, or down the street. It’s a fully digital world, right? You’re not sitting with people very often.

    Foster: Yeah, we’re relying on Slack, we’re relying on Slack calls, Zoom videos, everything to just stay in touch and work and collaborate with one another, yeah.

    Ashley: Steve—yeah.

    Howard: But I think that’s where—yeah, so, I was gonna say, I think that’s where those centralized processes that we’ve kind of all been working towards, luckily, for the last few years, are really gonna pay off. Because how do we not have that? If you imagine how things were maybe five years ago, it would’ve been a much bigger transition to get to where we are now, whereas it was almost like most of our organization was kind of ready to go. We were already there, just—you could call foresightful, but really, it was a little bit of chance, I think. Yeah, those processes have held things together remarkably well, and we’re very lucky for it.

    Ashley: You know, nobody would wish for something like COVID or a pandemic on any of us, but if you look back, you know, I think back when we were virtualizing things and developers were starting to be able to just create a fully self-contained environment to develop on their laptop—I saw people working on planes, right? First time I saw somebody writing code on a plane—that’s awesome.

    But so many things, and then of course, adding the cloud and all the services and software and tool chains and pipelines that we can create now, it—I mean, I think we’re lucky, if you were gonna have something like this happen, now is the time where we can actually continue to be productive, and put more emphasis or pressure on ourselves to deliver software. Do you agree?

    Howard: Yeah.

    Foster: Yeah, absolutely.

    Howard: Yeah—sorry, Stuart.

    Foster: [Laughter] We agreed at the same time, that was great.

    Ashley: [Laughter] 

    Foster: But you’re right, I think, like, some of it—you know, it’s a little bit accidental, but as you mentioned, setting up the development workflow to be able to do it on a laptop, people are shifting their hardware, they’re removing their hardware and going to the cloud, right? So, it’s kind of been like an expansion and a reduction, all—just naturally, but also, you know, you think about DevOps and there’s also, you know, cost savings to be had by not having to have this footprints, not having to worry about these things.

    So, this could be, in some ways, pretty revolutionary or changing for development costs for a lot of companies.

    Ashley: Steve, you were gonna jump in there—anything to add?

    Howard: Yeah, no, exactly the same. I think the fact is that we—you know, a lot of that infrastructure is being built up and the idea of being able to run our development pipelines kind of in a cloud environment or whatever and having that all ready to go just meant that we didn’t have to pick up ginormous work stations. I remember back in my days of developing in aviation software, we used to literally have, you know, Unix boxes stacked up on the side somewhere that we had to go and log in on each one and run our testing for each single one. It’s just unthinkable. It would never have been possible.

    Ashley: That would be insane today. You would never. I mean—

    Howard: Yeah.

    Ashley: – if you have a choice, you would never do that, right? [Laughter] 

    Howard: Well, exactly, and the very fact that, you know, 10 years ago, we still had to do that, how could we have transitioned? It wouldn’t have been possible. And so, it really is testament to how far we’ve really come in 10, 15 years to see where we are with our testing processes, the fact that we can do, you know, 99 percent of it, if not all of it can be done completely with no connection to the physical space.

    Ashley: Mm-hmm.

    Howard: It’s all—and that can even mean, you know, in the cloud completely, so it’s not even within our organization. It’s securely being done off site somewhere in the ether. [Laughter] 

    Ashley: Yeah.

    Howard: And that process is still completely manageable, it’s something we can all see as developers, everyone can get to see where we currently are with that process, you know, what the latest test run gave us across the whole—

    Ashley: I’m really curious—yeah, I’m really curious what your experience in market is. Do you see more people who are kinda doubling down and getting even more serious about their DevOps implementation and taking it to the next level, or do you see more people coming to you that maybe are relatively new to DevOps and are looking for tooling and looking for ways of helping them, you know, not just get started and learn all the problems themselves, experience them themselves, but looking from the experience of a software provider like yourself, or is it a little bit of both?

    Howard: I’d say a little bit of both. I mean, obviously initially, there was such a lot to do already and everyone kind of had no time to really think—understandably, but probably within, you know, since the end of the summer, maybe, things have started to come back and people are actually going, “Well, hang on a minute. We’ve got a process, but now, how can we actually make this a lot easier for ourselves?” and looking to improve that.

    And I think, you know, for a lot of the organizations where, perhaps, doing things off site would never have even been considered. And I think this was to Stuart’s point—that kind of, that’s been forced upon people and it’s almost gonna change the way we develop forever now as a result of that, because we have now, we’ve done it, we’ve experienced that it’s possible, and it’s safe and it’s secure. 

    How can we now make that better and make our lives easier by not having that massive footprint that Stuart was mentioning? We’ve actually got something that’s very, very lean, it’s in the cloud, we’ve got machines when we need them, but not when we don’t need them, and how can we really make that process as efficient as possible from the results side, you know, making sure we’re doing what we’re supposed to be doing as quickly as possible, but also actually trying to reduce our footprint on process and power, for example? You know, the development time that’s involved in that, how can we make that as efficient as possible?

    Foster: Yeah, and I also think what we’re seeing, too, is, you know, our existing customers, they’re doubling down on their tools, you know, because they need to rely on something, right? And that’s also, what it’s done is also, I think, is changing some of the roadmap items that all products are looking at, right? There’s a whole new—not a whole new paradigm, but you know, there’s a little bit more seriousness being taken in terms of some of these features and functionalities of being able to work remote, or the requirements for more cloud usage, right? We’re talking ease of use, we’re talking about things that are, you know, related to workflows to make working in these conditions—not just in these conditions. Assuming you get back to the office, just more efficient as well.

    Ashley: Mm-hmm.

    Foster: So, we’re seeing that at the same time, too. And then, you know, from the perspective of new customers coming in, they’re almost expecting these things to already be there, right? Because they’ve run up to these problems as COVID has created and they’re trying to find the solutions that are gonna be kinda turnkey out of the box.

    And also, I’ve done, I’ve been talking with various analysts related to our product, but it seems as though the analyst community itself is, they’re seeing the requirement for, you know, ease of use in DevOps becoming more important as well at the same time.

    Ashley: Mm-hmm. I think that may be also, in part, the expansion of DevOps into the organization. They may have had a core group of teams that were really taking the challenge of DevOps and implementing it and now, how does the rest of the organization do the same thing, but accelerate their adoption of it?

    Foster: Yeah, and I think also, a lot of development groups within a single company—you know, you may have a security team, you may have a development team, you may have a LiveOps team—they’re starting to recognize that there are choke points in the development processes.

    Ashley: Mm-hmm.

    Foster: And so, how do you unblock that? How do you make it more easy? How do you offload some of that choke point onto a broader group, right? A lot of the idea of DevOps, DevSecOps, whatever you wanna call it, right, is that developers are owning the code that they produce in more broad ways. That’s writing unit tests, that’s making sure they do the QA on it, right? That’s when they’re, you know, they have to implement the security requirements then and there, right? They need to ensure the traceability is done, right? They have to—they’re taking that into their hands. It’s becoming not so much that there’s a DevOps group that magically handles all this infrastructure, it’s becoming part of each developer’s responsibility as well, right?

    Ashley: Hmm. Interesting. You know, I know, as you scale things out and you adopt new tools, tools do a lot of things for you, but there’s oftentimes, there’s approaches that you need to adapt into your tools, things like what are our coding standards, what are our testing, what level of testing do we do? How do we break it into chunks? When do we do security testing of our code?

    And that’s part of that learning curve to is, how do you adapt a tool set to both what your requirements are, but also, you would like it to kinda take you to the next level. What are some of the things that really can help people kind of propel their expertise in the organization’s ability to deliver software more effectively?

    Howard: Yeah, that’s a really good point. So, one of the things, you mentioned coding standards there, but obviously, you know, those can come from multiple sources, they can be something that’s actually being required by a standard, for example, if it’s in a safety critical, security critical world. It could be something that’s actually demanded by the organization itself—they have their own rules or there’s certain things in any particular project that, if you do things in a certain way would make, you know, it would be dangerous or insecure or something. So, they have some controls, as well, of that.

    And what we do see is, the more that we can get that information out to our developers, for example, the more value that has as well. So, yes, we can help enforce coding standards, for example, with our Static Analysis tools. But that’s only half the story. Enforcement is one thing; actually getting the developers to understand what the problems are as they’re creating them and to have easy access to the help and the quick feedback of those problems makes it much more powerful, because it gives them, (a) you know, all the resources they need to deal with the problem quickly and to not perhaps find it a challenge to have to deal with, but also, they learn. 

    And so, the next piece of code they write doesn’t generally have the problem. So, there’s a developer learning element there as well, which we can control centrally, and then spread out to all of these remote workers working—you know, without that team collaboration that we used to have, we’re still able to do the same things. So, that’s become something that’s really focused people’s minds on the capability there in terms of what that means. And as I say, that applies to security standards that are well known or requirements for quality that we have to work to for a security or safety standard—

    Ashley: Mm-hmm.

    Howard: – but also the company’s own rules.

    Foster: Yeah, and I think, you know, outside of just simply the tools, development teams are, you know, taking a step back and re-evaluating or thinking more about what their software development process is, holistically, right? You know, Steve mentioned there some of the tools, best practices, but you know, you also have to think what kind of, you know, software development methodologies might you be using. You know, sometimes it doesn’t matter whether it’s agile, waterfall, whether you’re using Scrum or Kanban, you know, I think it’s giving people a moment to think about what’s the most important kind of Dev process that is gonna fit this team or how you have to work now, how it fits the businesses. 

    And then implementing processes to ensure the organization, communication, and what systems need to be put into place to help prevent any problems that arise during development and kinda identifying and understanding that. Then you’re picking the tools, you’re building your development workflow, whether it’s in environments, it’s setting up DevOps pipelines to ensure that everything’s followed correctly. It’s the tools, but it’s the process, and that’s changed, too.

    Ashley: That’s interesting. You know, we’ve done some research about the influence that developers have had on the organization, too. Even outside of development teams, you’re seeing DevOps and agile kind of principles, even though they aren’t calling it that—daily standup meetings, shorter intervals of deliverables, more kinda asynchronous communication tools and collaboration kind of tools, which really makes it easier, I think, for the development teams, the product teams—you’re a product manager yourself, Stuart.

    Foster: [Laughter] 

    Ashley: And beyond—operations, security—everyone is kind of forced to work a little bit differently, but also can benefit from what each other knows about how to do that more effectively. And we seem like we’re at this sort of serendipitous time of being able to leverage that, which fits right into the tooling approaches that we can take.

    Foster: Yeah, and you know, when you look at it, like, it always just ends up communication. Whatever the communication is, whether that’s the workflow, whether that’s in-person, whether that’s meeting, whether that’s documentation, it’s all about communication, and that’s true not for just development, but for business, right? Developers working, you know, with other groups—marketing, sales—everything, it’s all communication. It just keeps business running. And that’s kinda the funny kinda perspective that may not necessarily be outright visible by simply saying, “It’s communication.”

    Howard: What’s interesting is, that’s also kinda mirrored in the tooling, because a lot of the work we’ve been doing recently is about how to make sure the tools have all got open interfaces to communicate with the next thing in the pipeline or to feed that data back up to some dashboard that’s gonna be shared with the rest of the team.

    So, yeah, it’s kind of an interesting point, but that also applies with the tooling that we’re trying to build around that.

    Foster: Mm-hmm.

    Ashley: Mm-hmm. A really great point. So, to give you fair warning, I’m gonna ask you your three recommendations on best practices in the current climate, so be thinking about that.

    I was thinking about your comment, Steve, about being able to better integrate tools together and then taking the data from the tool sets, the pipeline of the process that we’re going through and the pipeline of work—there’s so much value in that data that sometimes ends up as exhausts or on the cutting room floor, right, to use two different metaphors. But that information could be extremely valuable, putting it into dashboards, feeding it into continuous improvement processes, shortening the window for getting feedback for, “Here’s a security vulnerability or a coding standard” or some kind of an error that you can fix in minutes instead of days and weeks and sometimes months, right? The security audit at the end of the release is much easier, because you’ve already gone through so much to it. Those can all be things that can help us.

    So, I gave you a little bit of time there to think about your three best practices. I did my best job of stalling that I could. So, who would like to go first? What would you recommend people consider today?

    Foster: Yeah, I’ll go. Just, I want to reinforce that message of,  you know, figuring out how you’re going to work as a development team, develop that communication, right? And that’s using your tools.

    I think in the world of remote work from home, having kind of a common language—again, back to communication—of how, you know, each developer’s system is set up. So, common environments, they should be using as close to the same setups as possible so that, you know, when one of you is running into an issue, it’s easier to get that information across, right? You’re not gonna have to install something obscure because somebody’s using something obscure. You should all be trying to talk and use the same tools so you’re all speaking the same language. That allows you to have that velocity that you could otherwise lose.

    And in this world, too, security hasn’t been more important than ever—security is more important than ever. So, think about securing your Dev processes, developing secure software, it’s gonna be very important. And those are kinda my three.

    Ashley: Okay, great. All good ones. Steve?

    Howard: Yeah, I’d answer—so, my three would be, first of all, to automate; take the process away from the day to day effort of the development teams and organizations around them. So, yeah, that’s the first one.

    Report—so, that’s the communication piece, I guess, but get the information to the right people at the right time and hopefully in the right sort of time window.

    And then shift left. So, then, we make it more efficient. Then we look to where we can try to remove the fat in the process and take away the time that was wasted and try and move it to the right position in that process. So, yeah, those three, in that order. [Laughter] 

    Ashley: Great. Excellent, excellent. I’m curious what you both think will be some of the things that will stay with us, the things that’ll last past a forced work at home, forced work remote. Maybe people are returning to the office, maybe they aren’t. What do you think are some of the things that will stick with us? Because it wasn’t like we were doing this for two weeks and then we’d just go back to the office—this is an extended period, we’re developing new habits, we’re changing how we’re working. Why go back, sometimes, right? What are your thoughts?

    Foster: I think you’re gonna see a lot of these workflows stay around. I think that the reliance on the numerous tools to keep velocity up and reduce and automate are just gonna stick around. You know, you don’t want to be wasting time on trying to get something to work. It’s easier to just simply automate it and get it out of your hands so that, you know, at least from a development perspective, you can just focus on what’s important, which is, you know, coding, the feature, getting the product out the door, right?

    Those are gonna stick around, and I think the tools, over the course of time, are just gonna keep reinforcing and strengthening that kind of automation pipeline.

    Ashley: Good.

    Howard: Yeah, no, and I think, from my perspective, just to add the one small piece there is, the cloudification, essentially, of the processes. So,  I think, you know, people will stop having the premises to host great, big server racks in, and that’s just all gonna move off site, off premise, outside of the organization, for sure, yeah.

    Ashley: Yeah. To kinda add what you both have said, which I think supports your thoughts on this is—you know, as we change to whatever we’re gonna go back to, I don’t think it’s gonna be back to what we were doing. But it’s gonna be a hybrid, right? I can’t imagine everyone shifting back to a different way. It’s—some people are still, maybe even more percentage of our teams are working remotely, never going back into an office, but we may have some that do.

    We’ve broadened the reach across the globe of people that we’re working with, the product and vendor, tool vendors that we’re working with. Those things aren’t gonna change, right? 

    Howard: No. No.

    Ashley: We’re gonna stay doing that. If we’ve found something that works better, you’re gonna stick with it, right? So, you’re gonna stick with—you’re gonna stay in the cloud, push more in the cloud, right? And leverage some of the automation, the processes that you talked about, Stuart.

    Well, this has been really fascinating. Would love to have you both back, have you guys back and we can talk some more about—I’m gonna delve into static code analysis evangelist, your role, Steve, get into some of that, which I know you all have some of your products do and to security and to some other areas within the software pipeline, but this has been really great. I’ve really enjoyed talking with you both.

    Howard: Yeah, it’s been a pleasure.

    Foster: Great. Thanks, Mitch.

    Howard: Bye bye.

    Ashley: You bet. We’ll see you soon.