devopsy.kinsta.cloud

Tag: shift left

  • Get Comfortable with Shifting Right to Improve Resiliency

    Get Comfortable with Shifting Right to Improve Resiliency

    The operations team receives a page at 2:00 a.m. Something isn’t working correctly. Do they know how to fix it? Perhaps more importantly, do they know whether it matters to customers or how it impacts the business? There are different techniques that companies implement to maximize the quality of their software. We talk about shifting left or right, chaos engineering and others. While they each have their own goals to improve quality, there’s an underlying common goal as well: All three techniques increase your team’s knowledge of the system and their ability to resolve problems quickly.

    Here, we’ll walk through the software life cycle and discuss the common challenges to software quality, running complex systems and managing dependencies between teams. The common thread is a proper observability strategy which can help software quality, hand-offs between different teams and resolve problems quickly by ensuring whole-system resiliency from design to maintenance.

    Techniques for Improving Quality

    Shift left is a concept that has attracted much attention recently. The practice revolves around the idea that detecting and correcting problems sooner minimizes their impact. As the thinking goes, they’re easier to fix, they wind up costing less and there are fewer ongoing issues with bad code.

    The rise of DevOps and DevSecOps has propelled the popularity of shift left. Yet, there’s a fact that development teams often miss: no matter how early an organization spots errors, problems will inevitably remain. Early testing can’t catch everything. As a result, some organizations have begun to rebel and move to a shift-right framework. It normalizes errors and aims to better resolve issues across the development process, including post-production.

    Shift right is the perhaps less-extreme cousin to its rebellious relative, chaos engineering. With chaos engineering, team members cause problems on purpose to keep everyone on their toes and test the limits of the system. They shut down servers, disconnect wires, delete files, etc. The goal is to confirm system resilience–from a user perspective, there should be no impact even with all of this chaos. A resilient system means that teams can fix failed components at their leisure during the day. With a resilient system, critical failures are less likely to happen and it becomes second nature to fix them when the dreaded 2:00 a.m. page does occur. Unlike chaos engineering, shift right doesn’t explicitly break things–it just makes the assumption that broken things will exist so teams need to be ready to handle them.

    Both concepts (and yes, even chaos engineering) are valid—but each company needs to carefully analyze which is right for them. One variable to consider is the complexity of the system you’re building. Working on a rocket launch or a surgery robot? Probably should shift left as much as possible. Changing the color of a button? Maybe it’s not the end of the world to verify the fix in production. For most of us, projects fall somewhere between these two extremes–so sometimes shifting left is necessary, and other times shifting right is pragmatic. A careful analysis of the risks should be performed to determine what kind of testing is needed.

    Although teams tend to be pretty good at analyzing the complexity of their software and making development process decisions, they often forget the complexity of other companies’ code which they depend on. Even a simple web application deployed today has tons of dependencies: The server it’s running on, the local ISPs, transit ISPs, load balancer, DNS provider, TLS certificate, CDN and browser, just to name a few! After analyzing the complexities of all of the dependencies across the internet, it becomes clear that finding every possible issue that might impact their application is a daunting task.

    The goal, again, is to find any issues before they reach production. Or if they reach production, fix them before they impact customers. Or if they impact customers, fix them as soon as possible after that. To do any of these things quickly, the level of observability of the overall system is paramount. With simple systems—or even complex systems that are entirely under one company’s control—tools such as application performance monitoring (APM) are common. But when the scope of the system is as big as the internet, internet performance monitoring (IPM) tools are necessary instead.

    With both APM and IPM tools, the goal is to know as much about the system as possible – enabling teams to identify problems quickly, pinpoint the root cause, and fix it. When an organization is equipped to monitor development and operational processes effectively from application to delivery, there’s a shift toward higher quality code, fewer errors, improved security and lower costs for the enterprise. There are also fewer disruptions and distractions for engineers, DevOps teams and others.

    Dealing With Disconnects

    Whether shift left or shift right is used, there will be problems that enter production. There is an invisible wall in many organizations–where engineering teams “throw code over the wall” to the team which is now responsible for running it. In some companies, this is the operations team; in others, it’s site reliability engineering or production engineering–the common trend is that in most companies, it’s not the same team who created the software.

    These operations teams often lack the nuanced information they need to resolve production issues. They might spend hours or even days attempting to track down the source of the problem. There’s often finger-pointing between different teams who created or own different portions of the system. Things can quickly devolve into chaos—and tempers can fray. After all, these kinds of problems always happen at 2:00 a.m., right?

    That’s where end-to-end observability enters the picture. Imagine a framework with proof that the development team has performed sufficient knowledge transfer to the operations team. That the operations team has asked all of the relevant questions about running and maintaining the application. Using the right tools, engineering and operations teams need to monitor every aspect and every component of the system. Creating this monitoring strategy means that everyone involved also builds an understanding of the system–everyone is on the same page. All of a sudden, there is a common language to discuss problems, and teams operating the software in production can take debugging a lot further before they need to ask for help.

    Ultimately, shifting left or shifting right is a personal choice for every organization. Even if every possible test case is executed before deployment to production though, issues will happen–so take the opportunity to implement a robust system-level monitoring strategy that can be used to solve technical questions during that 2:00 a.m. call.

  • Shift Left Testing in Microservices Environments

    Shift Left Testing in Microservices Environments

    By now, it’s common knowledge that the later a bug is detected in the software development life cycle (SDLC), the longer it takes and the more expensive it is to fix that bug.

    In 2017, the Ponemon Institute found that it cost around $80 on average to fix a defect detected early in the SDLC and around $7,600 to fix the same defect if it was detected in the production stage.

    Relative Cost of Fixing a Defect Based on Development Stage

    Design

    Implementation

    Testing

    Maintenance

    Cost of Fixing Defects

    1x

    6.5x

    15x

    100x

    While these numbers can be debated, it’s clear that early detection of defects can save organizations significant time and money while enabling those organizations to deliver higher quality software more rapidly. For these reasons, many organizations are implementing shift left testing practices, including in microservices environments.

    Shift left is a set of practices intended to find and prevent defects early in the software delivery process instead of during the testing and maintenance stages. The idea behind shift left testing is to improve software quality by making actionable test signals and feedback available as early in the SDLC as possible.

    Here, we’ll highlight how shift left testing, in combination with test automation, can better enable both innovation and fast delivery of high-quality software. We’ll also address some of the challenges involved in implementing shift left testing in microservices environments, as well as introduce a valuable strategy for successful microservices implementation.

    Shift Left Testing Benefits

    The waterfall model of development involved the explicit passing of responsibilities between highly specialized design, development, QA, and release teams. It also involved lengthy feedback loops.

    Scrum and agile methodologies made the entire SDLC more flexible and nimble by introducing sprints and allowing more frequent iterative development and delivery. Further, DevOps and DevSecOps focus on removing the silos between development, operations, and security through tooling and automation. As a result, the time to market and quality have improved dramatically. Adding shift left testing into the mix better positions teams to handle the broad range of responsibilities from the design stage through the maintenance stage as effectively as possible.

    Shift left testing focuses on prevention rather than detection. Shift left benefits include the following:

    • Increase efficiency by eliminating bugs earlier in the SDLC.
    • Reduce human errors and associated costs.
    • Increase delivery speed and reduce the time between releases.
    • Improve the quality of software.
    • Gain a competitive advantage.

    It may be helpful to think of shift left testing like this: Test early and in parallel with development, instead of the traditional waterfall model of development followed by QA testing. 

    Shift Left Testing Challenges in a Microservices Environment

    By structuring an application as a collection of loosely coupled, independently deployable services, a microservices architecture enables frequent delivery of large, complex applications.

    While shift left testing practices bring many benefits, shift left testing also comes with three challenges that organizations must address when implementing it across a decentralized, microservices-based stack:

    1. Building and managing each microservice separately reduces development time. However, implementing shift left testing at the microservices level means shift left practices must be applied to each individual microservice.
    2. Microservices developers can choose the programming language and service dependencies best suited to their development needs. However, as a result, shift left testing practices need to be implemented and managed across different programming languages and dependencies.
    3. To ensure that microservices function as expected, teams usually run integration tests, end-to-end tests, and performance tests that assess functionality in the context of the larger stack. Shift left testing is especially challenging in the microservices context because of the complexity of testing against the stack.

    Microservices, each with their own SDLC pipeline

    Pipeline Scaling Considerations

    Several classes of automated shift left tests can be integrated into the DevOps pipeline, including unit tests, integration tests, performance tests, and security tests. Different microservices will have different testing needs.

    The key is to apply shift left testing practices to each microservice while maintaining the flexibility and decentralization of those microservices. The ideal workflow system and DevOps pipeline implementation must simplify pipeline creation while enabling development teams to manage the pipeline and requiring minimal input from DevOps and platform teams. In other words, shift left practices must be tailored to the specific needs of the microservice in question to make the testing process scalable and easy for development teams to manage on a continuous basis.

    As just one example, GitHub Actions and similar CI/CD pipeline tools enable development teams to use GitOps-based practices to configure, manage, and administer the entire pipeline for their microservice while leaving infrastructure testing and testing that relies on or affects other parts of the system to DevOps teams.

    Test Environment Considerations

    Test environments are environments where unit tests, integration tests, performance tests and security tests can be run on an isolated collection of microservices and configurations. The nature and quality of these test environments directly correlate with the effectiveness of shift left testing.

    A cloned production or staging environment can be used for testing when the number of microservices is low and overall complexity is relatively low. However, once an organization reaches a threshold number of microservices or a threshold level of complexity, runaway infrastructure costs and operational burdens make such environments untenable.

    In response, some organizations rely upon smoke testing (minimal viable stability testing) and mock-based integration testing early in the SDLC. While such tests can prove useful in the near term, deferring essential testing to a later stage in the SDLC often introduces test signal and validity problems in the staging environment and can lead to delivery delays and serious issues with software quality.

    So how can shift left testing be used successfully in a microservices environment? The solution involves the use of scalable, ephemeral test environments.

    Ephemeral Shift Left Testing Environments

    The availability of high-quality test environments is a common bottleneck for organizations looking to scale shift left testing practices. Using a few high-fidelity staging environments for signal testing can slow the release process. More to the point, it tends to slow down the entire SDLC and increase stack instability over time.

    Ephemeral test environments offer a fundamentally different approach. These sandbox environments use isolated environments that enable users to test independently without impacting other users. In particular, they enable isolation while sharing resources (e.g., external cloud dependencies and hosted databases) using application-level multitenancy instead of the resource duplication approach in traditional environments.

    The idea is that microservice test environments can be used to test changes against a much larger stack of services. Because multiple tests can be conducted at the same time, test coverage and resource efficiency vastly increase.

    High-quality sandbox environments can spin up in seconds and scale to tens of thousands of microservices. In fact, each pull request, branch, or commit can have its own test environment without significantly increasing infrastructure and operational costs. Any change, no matter how small, can be tested in isolation to ensure performance and stability against the entire stack.

    Shift Left Testing Solutions for Microservices

    A microservices-based organization looking to scale shift left testing across the organization must first address several challenges. Considerations include the need to apply testing practices to separate microservices, the need to accommodate different programming languages and service dependencies and the need to test against the stack.

    The right tooling and automation are needed before shift left testing practices are scaled across the organization. In addition, a cultural shift may be required in order to help developers understand the ways in which shift left testing can expedite and improve the core functionality that they are developing.

    To apply valuable shift left testing practices in a microservices environment, organizations should consider using ephemeral sandbox environments. Using the right set of tools in a sandbox environment can enable development teams to test microservices throughout the SDLC. In doing so, organizations can ensure those services follow security best practices and perform exactly as intended, both independently and against the stack–all while reducing costs, speeding time to market and improving software quality and user satisfaction.

  • Implementing Shift Left Security in the Cloud

    Implementing Shift Left Security in the Cloud

    While ransomware has been the leading concern for enterprise security teams over the few past years, software vulnerabilities are nipping at its heels. The boom in cloud-based apps and services and increased digitization of work have been a boon for hackers, who are taking advantage of developers’ and DevOps teams’ attempts to work faster and smarter to keep up with demand. One estimate says that four out of 10 zero-day attacks carried out in the last decade happened in 2021 alone.

    Many things account for this increase. Developers are stretched and are reusing code, which allows for misconfigurations and vulnerabilities to reappear unexpectedly in different programs, and the use of multiple cloud services fragments security measures and reduces visibility into the code running many enterprise functions. This is why developers and security professionals alike are paying more attention to security throughout the software development life cycle (SDLC), particularly in the early stages.

    Shift Left Security Principles and Challenges

    The zero-day surge has led to an increased interest in shift left practices as a way to make security a priority in the development process. Shift left culture brings security into the equation much earlier in the software life cycle, before the software is deployed, rather than patch bugs after users report them. This preemptive approach helps head off vulnerabilities that can affect an application’s security posture unbeknown to its defenders.

    Shift left principles can also enhance security when developers build applications for cloud platforms—such as Amazon Web Services, Microsoft’s Azure or Google Cloud—where visibility into the proprietary code and security tools of the platform can be limited. In a shift left culture, DevOps embeds least privilege policies as part of the daily work on cloud workloads, to protect network infrastructure and avoid granting excess permissions on those workflows.

    For example, setting up role-based access control (RBAC) on Kubernetes containers enforces a least privilege model on those clusters and avoids excessive permissions that can lead to a breach, while removing admin credentials from continuous integration/continuous delivery (CI/CD) workstations shuts off hackers from using those pipelines as an attack vector.

    Shift left security is a great concept, but runs into hurdles in execution, mainly the internal conflict between development teams who want to move at the speed of business while security staff takes a more cautious approach. This leads to friction, with developers warning about access denials affecting their work and causing deployment delays, while the security analysts worry about data breaches caused by too many users with unchecked admin privileges.

    Working out this friction requires a mindset shift, where DevOps understands that security is part of the development process, and management can trust that developers have security in hand. This also requires tools and training to integrate shift left practices, such as platforms that can bridge the security needs of the CISO (by granting greater visibility into the network infrastructure code, for example) and the agility needs of the DevOps teams (by automating security guardrails and entitlements, for example). The ultimate goal is creating what is known as a DevSecOps model, bringing together development, security and operations into one agile, safe and efficient workflow.

    How to Adopt Shift Left Security on the Cloud

    Four best practices can help:

    ● Gain deep visibility: With enterprises often working in multiple cloud environments—and mixing public and private clouds into hybrid infrastructures—visibility into assets typically suffers, which makes getting a clear view of risk difficult. Being able to discover all identities, permissions, configurations and resources in the environment and all the access paths to specific resources can help build a contextual inventory of cloud assets to better manage them and carry out policy analysis.

    ● Prioritize risk: Risk management is a key practice of any security program, but the flood of alerts from multiple cloud providers and systems can overwhelm security operations centers, which are often short on talent and time. Automation tools can now search and prioritize risks across an entire cloud environment, from development to production, spotting problem combinations and alleviating the workload of sorting manually through alerts across multiple silos, freeing the staff to work on higher-priority mitigation and remediation tasks.

    ● Just-in-time access: Zero-trust architecture has become the standard to aspire for most security defenses, but it is often challenged by the difficulties of managing access privileges. Just-in-Time (JIT) access, granting limited-time revocable rights, is a useful tool for building a zero-trust security architecture and enforcing least privilege policies to maintain it. Technology tools that actively manage and monitor developer access to cloud environments, including a strong audit trail of privileged activity, give security a means to enforce JIT access and zero-trust in turn.

    ● Find and prevent vulnerabilities and misconfigurations: Infrastructure-as-code (IaC) has made software deployment in the cloud (much more agile and efficient with the use of virtual machines, containers, microservices, etc. However, security is often an afterthought both pre- and post-production when standing up cloud infrastructure. Organizations should scan code for security policy violations and other risks in order to mitigate flaws before they reach production and to detect and fix problems in production environments. With feedback and automated guardrails built into development workflows, DevOps and security teams can reduce security risks in their code.

    Like zero-trust, shift left is on its way to becoming a standard practice for cybersecurity assurance. But building a DevSecOps model takes more than good intentions. It requires practices, tools and training to execute that model, and a mindset of partnership across the enterprise and with outside cloud and security partners. Only a group effort will provide the cloud security assurance an agile business needs.

     

  • Shift That [bleep] Left

    Shift That [bleep] Left

    Seriously. The more you can shift things like security and test to the left, the more responsive the DevOps process will be. Some things are better shifted right … But only while solving the problem with shifted-left work. A good example is blocking zero-day attacks proactively while the development process fixes the code to stop the attacks permanently.

    One thing we in IT are terrible about is following up on that “fix it” part—because we are very busy. Once a problem is blocked, we tend to fight other fires because we have too many things going on. We have absolutely got to get better about that. But I don’t want to get sidetracked. Today’s topic is about fixing bugs/vulnerabilities as they are created.

    One of the issues with the way we used to handle test and security was that by the time we had “This is an issue we need to fix,” feedback, the developer was off doing other things. But with the increasing inclusion of security and testing in the DevOps process, we now have the ability to provide feedback far more quickly.

    My favorite innovation in the DevOps space is not some wild new bug/vulnerability detection feature, nor is it rampant use of AI (though some uses of AI do come close to being my favorite). It is the humble IDE plugin.  As a developer, writing buggy or insecure code is never the goal, but it happens every day. IDE plug-ins, hooked into test and security software, allow developers to get a heads-up that there might be an issue while writing the code. This not only addresses the problem while it is fresh in the developer’s mind, it also allows developers the opportunity to improve their craft simply by understanding which issues they create often and learning how to avoid them.

    We have all written bad code. I once actually over-wrote databases for a bank; it was a great humbling and learning experience for a young hotshot who needed to be reminded he was not perfect when coding. Others no doubt have similar, if less catastrophic, stories to tell. The ability to get warned about issues—but quality and security—while writing the code cannot be overestimated. Some developers will hate it because it is more intrusive than the old school “Okay, all the code is written, you can test now,” process, but it is powerful precisely because it is more intrusive and proactive.

    The challenge for those in the testing and development security spaces will be to keep false positives at a minimum. I’d say zero, but I’d rather get the occasional “Oh, that. Yeah, not an issue in this case,” than go through having it tell me things throughout development and still end up with buggy/insecure code.

    You all are still rocking it. The amount of software in use every day across the world is staggering, and you’re cranking it out and keeping it maintained—whether you’re working on internal applications or commercial packages. You’re making it work. Take a look at QA and security IDE plug-ins, make an honest assessment of whether they could make your work more secure and/or higher quality. I think they can for nearly all of us. And they can save time by doing it earlier in the development process. And maybe you’ll get recognized for rocking it. Okay, probably not, but it is possible.

  • Software Quality is the Heartbeat of the Best Organizations

    Software Quality is the Heartbeat of the Best Organizations

    JPMorgan recently announced it was hiring 2,000 engineers, despite the gloom in global economic markets. Is this not an odd risk for an organization to take, given the demand for (and cost of) software developers today? What’s happened?
    JPMorgan’s hiring drive is no mystery. Virtually every company in the world is leveraging software, with many transforming themselves into software organizations to drive new efficiencies, widen profit margins and create new markets. Software has been disruptive, so much so that the current era has been dubbed the “third industrial revolution.” Consumers are benefiting from switching providers at the click of a few buttons and most consume services through software. Bad software experiences today mean rapid customer churn and competitive disadvantage.

    Quality Should be the Heartbeat of any Organization

    In spite of quality’s importance, testing is still perceived as a secondary discipline and a check-the-box exercise. This reflects practices developed for assembly lines during the original industrial revolution. A product gets assembled and testers verify its quality as it leaves the production line. Testing’s goal in this context was to identify problems before they reach end-users. This same approach has been applied widely to software development and is still used by “waterfall” organizations.

    Yet, software development has been revolutionized by new methodologies and practices. Software today is constructed at incredible speed. Notice I’m using the word “constructed” and not “built”—the best developers leverage libraries or APIs to assemble software using existing components. They then add a few pieces of custom business logic to define the intellectual property of an application. Many organizations don’t even code software, instead customizing off-the-shelf software with vast configurations to modify functionality.

    The assembly-line approach to software development is no longer appropriate. When testing is left late, defects are found before a release or even in production. Release cycles are longer and products poorer quality. Yet, many organizations still stick religiously to this approach, with testing teams operating in distinct silos.

    The Role of the Tester is Evolving

    The best organizations in the world are changing how software is built. Testing is no longer a check-the-box exercise at the end of the assembly line. It is transforming into a discipline that revolves around evangelizing and driving quality throughout an organization.

    Quality should be baked in as early as possible, touching every artifact produced. Ideally, this should occur from the start, as a requirement is formulated, all the way through to the software being shipped. The idea of siloed teams working on different pieces is outdated. For quality to thrive, teams must collaborate.

    Modeling offers one approach to driving quality and collaboration throughout an organization. Its journey starts with a requirement. These are typically formulated by business analysts or product owners. Having been a product owner, I found that this usually involves lots of scribbles on paper before communicating to stakeholders on a whiteboard. The preliminary requirement then gets reviewed and refined.

    At some point, the requirement will be converted into walls of text in a project management tool. Modeling intervenes at this stage, adding visualizations that make requirements more consumable, less ambiguous and easier to understand.

    The word “modeling” may seem a scary concept, but it need be no more than a flowchart that encapsulates a requirement. The benefit of modeling is not only that it provides a digital collaboration asset; models can also generate coverage-focused tests before a line of code is written. Modeling creates requirements that are quality-driven and quality-driving.

    The Journey of Quality-Driven Requirements

    • Models Provide Data Upfront

    Developers today require rich data to develop, while testing is only as good as the available test data. Yet, many organizations still feed slices of production data into developers and test cases.
    This approach is problematic for several reasons. First, it often risks legislative non-compliance. Second, production data features large volumes of low-variant data. Data may not then be available when developing unusual-but-important scenarios. Imagine an overdrawn account with several beneficiaries: What should happen when a transfer is attempted?

    Modeling defines data explicitly in requirements. It allows engineers to understand a requirement explicitly, including the relevant data. From the modeled definitions, correct data can then be masked and subset, or generated synthetically. Data can further be delivered on-demand and at scale using database virtualization.

    • Models Provide Understanding

    Once the requirement and necessary data are available, code can be developed. Using text-based requirements, misinterpretations often lead to incorrect functionality. Visual models instead align closely with business logic, and we often see developers enriching models with additional context so that models align with the actual software implementation.

    • Models Generate In-Sprint Tests

    Once the code is developed, the same tests which were created earlier from the requirements can be executed against new functionality. This facilitates reuse and means the test cases are ready to be executed immediately.

    Some tests might further be selected for automation. In this instance, automation can be overlaid onto the requirements model, reusing code to generate scripts. These scripts then become part of the standard regression pack as new features are added to the product.

    A Change Request Comes In

    With iterative development, change requests are common. This typically creates havoc for quality, since it is difficult to understand the impact of a change, and time-consuming to update artifacts like tests and data.

    Traceability is key to overcoming this challenge. Models can connect to external DevOps tools to provide end-to-end traceability, from a requirement, all the way to code and tests. The impact of a change can then be identified directly. Models can be updated, automatically synchronizing test data, test cases, and automation to bring everything up to date.

    Testing for the Fourth Industrial Revolution

    Software is playing a vital role in virtually every organization, as already recognized by organizations thriving today. Baking quality in early is the only way to ensure quality software. The role of testing must change on a large scale. Models offer a way to achieve a “shift left” and collaborative approach to quality while testing earlier and more rigorously by automating and optimizing test and data creation.

  • Adopting Shift Left Testing in Software QA

    Adopting Shift Left Testing in Software QA

    I am often asked to recommend best practices for building software testing programs. The problem is that it depends on your definition of “best.” What works for an innovative startup developing software and deploying agile development sprints is not necessarily going to suit an established public company launching a new web product. The software development industry is so broad and diverse that there is no one-size-fits-all set of practices I can recommend. Put simply, the “best practices” are those that produce consistent results over time, meaning that they can be standardized. There has been a trend toward shift left testing in the past decade, and I have a few suggestions for getting the most out of development teams taking on this new responsibility.

    Shifting Left Testing Supports Agile Software Development

    “Shift left testing” means testing software for functional quality during the development phase, rather than waiting for the QA testing phase within the traditional software development cycle. Back when software was packaged for sale and deployment rather than delivered as a service, there was time in the development cycle for a dedicated QA phase. Developers started with a concept, analyzed their requirements and came up with a design before beginning to write the code. Once written, the software code would be integrated within existing systems (if needed) and then the software would be passed on to the QA team. Any bugs were sent back to development to be fixed and sent through the cycle again. All of these phases represented a conceptual move from left to right.

    Shift left is a departure from this cycle in which developers test their own software in an effort to speed up the process. This kind of development works particularly well for dev teams engaged in agile development. By having developers test their software as they go, bugs can be identified earlier in the development process, which saves time and resources and enables more thorough testing.

    Shift Left is Also Problematic

    There are some significant problems with the Shift left method that can’t be ignored. Specifically, developers are being asked to perform functions they were never properly trained to do—and often have no real interest in doing.

    Software developers are creative people by nature. Whether self-taught or formally trained, the majority of them have come to their positions in pursuit of creating something. Quality assurance testing, by contrast, is a destructive process. The goal of the QA engineer is to break things—kicking the tires and carefully documenting their destruction. And when you ask your creators to become destroyers, it doesn’t always go well.

    We can’t fault developers for not wanting to find faults in their code. Just as the best builders are not the best demolition experts, software developers want to create and then move on from one project to the next. So their reluctance to engage in testing is understandable. It’s part of who they are. The truth is that most developers find testing a monotonous chore. We need to learn how to encourage and incentivize them to become good testers of their own code.

    Provide an Encouraging Environment

    The first step toward making developers better testers is to educate them on the importance of testing as part of the development process. They may initially think that you’re only interested in saving money on hiring QA testers, or that they are being asked to perform the work of two employees. Introduce shift left testing as an integral part of the development process, explaining that continuous early testing is key to building high-quality software. By putting testing within the context of elevating software quality, developers will become more engaged in the process, raising the stakes and their involvement in projects.

    Make Software Testing Easier for Your Developers

    Since testing is not likely to be the primary interest of your developers, simplifying the task is going to make it at least more palatable to them. Test automation tools can be a big part of this strategy. Automated test scripts can be run flawlessly, frequently, and with much greater load capacity than with manual testing. You’ll be able to scale your tests to mimic a single user, or thousands of concurrent users, to discover how your software holds up under high demand. Pixel-for-pixel aberrations and errors in your graphic user interface (GUI) can be analyzed more quickly and effectively than with manual testing.

    Automated testing also takes much of the monotony out of testing, preventing burnout and resentment among your employees. Writing scripts also has the benefit of being, inherently, a creative, not destructive, process. By integrating the running of scripts into the normal workday of the developer, and emphasizing their value in building the best software possible, testing becomes a valued task, rather than a mindless chore.

    Providing Other Incentives

    When you adopt shift left testing, you just might be asking your developers to take on a role that they don’t enjoy as much as coding. And they may never enjoy it as much. But that doesn’t mean there aren’t actions you can take. If you want to keep your most valued developers happy and encourage them to stay with you, think about providing other incentives that make the entire job package irresistible. Make your company a great company to work for. From the working environment to other perks and compensation, any way you make your team happy can help them manage integrating new responsibilities.

    Testing Tools Lighten the Load

    As test automation technology gets more sophisticated, testing tools that incorporate aspects of artificial intelligence technologies to assist in test script generation and self-repair will play a larger role in enabling your developers to focus on the more creative aspects of the job — addressing design requirements and functionality (the “what” and “why” factors) in novel ways, and leaving more analytical, procedural tasks (the “how” factor) up to the tools. As testing tools perform more of the monotonous tasks for us, and with greater accuracy, developers will have more time for the creative tasks that keep them happy. These technologies are evolving as we speak and will address many of the issues described above. But you might still want to consider buying some doughnuts for the break room.

  • Where Does Your Data Go?

    Where Does Your Data Go?

    One of the most interesting developments in security and compliance in recent years is the ability to follow a piece of data through an application from input to consumption and see each bit that touches it. For me, the reason this is so interesting is that it allows postmortems to actually determine exactly what was lost in a data breach. “Oh, that library was compromised? Okay, let’s see what inputs that library actually touches …” It’s so very cool, given the free-for-all that data has traditionally been in applications.

    I’m not a huge compliance fan for a variety of reasons, but this is one great result of mandating compliance. I don’t think that we would have this level of data tracking without compliance requirements driving it.

    So what can we do with this information? That’s the interesting part that I think we’ll see developing over time. If we know everywhere that a given piece of data is used, we have a powerful bit of development information. It is still early days for implementation–this is not easy stuff–but think about it. I could see a future where all redundant data access is eliminated. If we’re getting the same set of data from multiple sources, and it is being fed to the same internal consumers, then we can standardize the you-know-what out of it. A singular library to standardize formatting, a singular library to standardize look-ups, etc.

    For the most part, we currently validate and format on input. Like straight away as the data is brought into the system. That is a great idea for a system where we are uncertain where the data is going, and are sure that some attacks are coming through those inputs. But what if we discovered 10,000 more lines of code to maintain than is necessary? What if data tracing allowed us to eliminate that code by centralizing those functions? We would have to harden the routine doing the validation and formatting–because it will be getting dirty data–but no more so than we are doing literally everywhere today.

    If we can reduce source code by using a tool that is designed to improve security, is that not the ultimate shift left? The volume of source code that handles data entering the system is staggering, also. For enterprise systems, most of what we develop is actually data processing, no matter how convoluted it gets. The data store is the central repository and apps are simply accessing and modifying it. So shift left with regard to data is huge.

    Consider it. I know that you are all super-busy with other projects, but less source code lightens the load on everyone; take a look at data tracing functionality in products you already own (mostly available in vulnerability management and source code scanning toolsets), and see if it suits your needs. It might save everyone time and will help with some data-centric compliance requirements, too. That leaves you more time to make systems better.

  • ShiftLeft Report Reveals State of Application Security

    ShiftLeft Report Reveals State of Application Security

    A report published today by automated application security testing platform ShiftLeft found only one in three applications has an attackable vulnerability. The report also found organizations that prioritized their remediation efforts based on the level of actual threat are fixing 76% of those vulnerabilities within two sprints lasting 12 days, on average.

    Based on millions of scans performed in the last year using the ShiftLeft platform to identify vulnerabilities, organizations have reduced the number of vulnerabilities they need to address by 97% on a year-over-year basis, the report found.

    In addition, the report noted that the average median scan time is only 90 seconds, which helped drive an overall 68% increase in daily scans year-over-year. Overall mean-time-to-remediate (MTTR) has dropped by 37%, the report also found.

    ShiftLeft CEO Manish Gupta said given the increased number of vulnerabilities, it’s more critical than ever for organizations to focus their remediation efforts on those that can be exploited within their IT environment. For example, the report found that only 4% of all Log4j instances discovered in the IT environments analyzed were vulnerable. Rather than spending weeks looking for every instance, Gupta said IT teams would be much better off focusing their efforts on the instances that might actually be exploited.

    The challenge is that not enough organizations have a way to score vulnerabilities based on the risk they represent to the business, Gupta added.

    While there is a lot more focus on adopting DevSecOps best practices in the wake of a series of high-profile breaches of software supply chains, the ShiftLeft report made it clear that presenting developers with a long list of potential vulnerabilities is counterproductive. The bulk of vulnerabilities that are being discovered are not actually attackable, noted Gupta.

    That doesn’t mean application security doesn’t need to improve, but it is clear that a lot of time and effort is being wasted, added Gupta.

    In theory, at least, as the cultural divide been application development teams and cybersecurity teams continues to narrow, the overall ability to prioritize remediation efforts should improve. The first step toward achieving that goal is making it easier to identify what percentage of the application attack surface is really vulnerable. Otherwise, developers presented with a long list of theoretical vulnerabilities will continue to prioritize developing new features and capabilities versus remediating vulnerabilities.

    One way or another, vulnerability remediation issues will soon come to a head. In the wake of an executive order from the Biden administration to improve overall application security within federal IT environments, many organizations will adopt similar policies and processes. Developers, naturally, will inevitably either push back or do an end-run around processes they deem too onerous. Rather than allowing a stalemate to occur, Gupta said savvy DevOps teams will invest more time in identifying the critical vulnerabilities that need to be addressed before cybercriminals exploit them.

    The issue, of course, is that cybercriminals are getting more adept at discovering and exploiting those very same vulnerabilities.

  • At Some Point, We’ve Shifted Too Far Left

    At Some Point, We’ve Shifted Too Far Left

    Those of us involved in DevOps have a tendency to see the world with blinders on. It is rather easy to fall into the “If all you have is a hammer, everything looks like a nail” trap. We have used the phrase “shift left” with the attitude that this is the solution to every problem in IT.

    Just for clarity, it is not. That is as blunt and unambiguous a statement as I can make. DevOps is great and has helped in many areas, but it is not the only solution, and not every problem is a DevOps toolchain problem.

    In the end, much critical work in IT is not DevOps-related, nor can it be impacted by DevOps. I’ll mention two areas that are obvious and talk in-depth about one of them.

    First off, no matter how far down the Agile/DevOps trail you travel, the wagon you are riding in is separate and distinct from the DevOps process. If you have a data center, all that hardware is still, in the end, hardware that must be configured and managed. For the “But we’re 100% in the cloud!” crowd (who say that loudly and forcefully enough that I suspect they’re trying to convince themselves that this is a good thing)—in the end, you are still using a network and computers to connect. The massive work-from-home migration due to lockdowns made it abundantly clear that IT is needed to set up and support both. Some of the horror stories coming across ops and DevOps social media were … ugly. Employees don’t want to be—and should not need to be—IT specialists to get their job done.

    The other easy one is security. Yes, insert security into development and DevOps workflows. Yes, make operations staff more aware of security. Yes, automate security as much as possible because we’re chronically short of security staff. But, in the end, lots of security is external to DevOps. While Kubernetes security is absolutely DevOps, password management, for example, absolutely is not. And that’s just one of many, many examples.

    So I am going to ask (but not answer) the question: Where is the line? How far left is too far? The answer is yours to decide with regard to your organization and the tools/environment that exist in your organization. In a primarily cloud environment, the answer to this question is far different than in a primarily data center environment. It is rare that any organization be purely one or the other at this point in time, so there are similarities, but few exact solutions we can name.

    Security is akin to a cost center. It is actually insurance, but few orgs view it that way. And even if they did, it would not make much difference. Much as my father once declared that he and Mom were “insurance poor” and canceled half their policies, most orgs want to contain the costs of security whether they admit it or not. When ‘security merged into DevOps’ hits the ears of people trying to stretch budgets just as far as they can, bad things tend to happen.

    DevSecOps implements security where it should have been all along—at the development stage—but it does not replace other security, it just makes our apps more protected. There are cases where security testing hours are seriously reduced, but since testing in all forms is the first thing to go when staffing is tight, again, it is not replacing something that was happening—it is making it possible for us to do what we should have been doing all along.

    It doesn’t help that our toolchains have taken on lives of their own—at many organizations, staff are required to maintain them—because they, too, are apps that DevOps struggles to help. Once you’ve invested time and money in building that edifice, you want to use it as much as possible. So, many orgs want to shift left everything in IT, right into the toolchain octopus.

    You know your org better than anyone. You are the ones making it work day in and day out, in all of your environment’s glorious complexity. Start talking about what is too much; when adding a step in an already complex toolchain needs to be justified. Push back against everything needing to shift left (where appropriate, I’ll say again for clarity, DevOps has done amazing things for us, but too much of a good thing is our living nightmare and, historically, is tomorrow’s technical debt). Keep rocking it, and make sure you are rocking the right things in the right places.

     

  • Threat Modeling as a DevSecOps Practice

    Threat Modeling as a DevSecOps Practice

    Software engineers are always under pressure to build more software, faster.  At the same time, there is increasing regulatory and market pressure for secure software that meets users’ and regulators’ requirements for data privacy.  This dynamic often puts software engineers at odds with application security or product security teams.

    In fact, 81% of developer teams at large enterprises have launched applications that feature insecure code, according to one researcher, with half doing this on a regular basis. The firm argued that developers and security architects need a better relationship, and I couldn’t agree more.

    These technical teams are working towards the same goal, so it’s in everyone’s interests—particularly senior leadership—to create a more cohesive working environment by breaking down barriers between teams. If the security of a system is designed and enforced by an external security team—disconnected from the rest of the engineering team—then this only reinforces security as a separate silo. But companies can ease tensions by embedding the responsibility for security in the engineering teams themselves: This is the thinking behind DevSecOps. Thinking about security starts when engineers think about solving the problem: At design time.

    While the shift left movement was born two decades ago with the goal of ensuring product testing happened early in the development stage, I believe we need to start left to make real change. In adopting this culture, security and privacy will be less of a bolt-on afterthought and a built-in fixture within the development process, ensuring that engineers and security teams alike are aligned and have visibility of each other’s objectives. 

    By reimagining the build, test, fix, build, test, fix cycle through building an architecturally secure design in real-time, technical teams can detect software vulnerabilities sooner rather than later, which means more efficient product deployment. This is where the process of threat modeling as a business tool can be of great benefit.

    Historically, threat modeling was a static, slow and manual process conducted on whiteboards, as part of a workshop led by a third-party security expert who lacked institutional knowledge of the inner workings of the company. However, this has evolved dramatically into an easily implemented, automated security practice that can be baked into the development and security cycle from the outset, empowering organizations to start left.

    Through automated threat modeling, the security workload burden is relieved for both security architects and engineers. Companies are, in turn, better positioned to keep pace with the cadence of ongoing software rollouts, as an automated platform suggests security improvements as an application evolves. The launch phase bottleneck that’s typically created during the testing stage is therefore removed, as security requirements are identified prior to development—which means that risk is managed before a line of code is even written. Based on this, it should quickly become apparent what a time- and cost-saving exercise threat modeling is.

    No developer or security specialist wants a flawed product to hit the market, so it makes good business sense to adopt a new approach that will lead to better outcomes. Enterprises that introduce threat modeling as part of their development and cybersecurity processes will be on course to embed a true DevSecOps system within the company, transforming the relationship between these two departments.

    In my experience, culture and growth are the key factors for enterprises that decide to adjust their cybersecurity diet to include automated threat modeling. Convincing the app dev teams to invest time into the process is, by far, the most significant cultural hurdle to overcome, because they need to witness the benefits—and up until this point they’re disconnected from the AppSec process.

    Once equipped with the tools and knowledge, delivering secure design will be second nature to developers and a new collaborative culture will be created. In essence, the more teams are given oversight of and accountability for software design flaws, the easier it is to recognize how vital these preliminary checks are.

    Likewise, there comes a stage of business growth when the ‘security first’ and ‘start left’ mindsets are essential to keep enterprise project management on track—and this ties into the culture. As the rate of scale within a company begins to increase and the pace of software deployment must quicken in line with customer demand and market competitiveness, discovering flaws at the end of the design process just isn’t an option. 

    With developers and security teams responsible for vast amounts of code, product resilience on delivery is crucial and ensuring a robust DevSecOps culture is the route to success.