Tag: software

  • Break Open the Black Box of Software Development

    Break Open the Black Box of Software Development

    If you asked an airline executive what their biggest expense was just a few decades ago, they would’ve easily answered, “Jet fuel.” These days, you’d be hard-pressed to find an executive in any industry that wouldn’t answer with, “Software.” But here’s the difference: With jet fuel, that executive would have told you that they can fly x number of seat miles per gallon/liter of fuel (or pounds of fuel, if we’re being technical). With software development, the expense is becoming more material–and far more variable–but the return on that investment is rarely clear. Over the past decade or more, software development has readily assumed its position as the fastest-growing (and often largest) expense line item in virtually every industry. All companies are adding software experiences to or around their products, and they need the people and tools to build, support and continuously optimize those experiences. And that costs money–a lot of money.

    The problem is that the language, the metrics and the world in which software development exists do not translate into operational or financial terms that business leaders can readily understand.

    “We invested X million dollars into software development last year,” a CFO might say. “And what do we have to show for it?”

    Meanwhile, a technology leader might respond with, “We’re doing X deploys a day!” or, “We wrote X thousand lines of code!”

    They might as well be speaking different languages because, in practical terms, they are.

    And the problem is, that you can’t manage what you can’t measure. To continue our airline theme, software development continues to be a ‘black box’—this hidden-in-plain-sight source of information—that financial executives can’t seem to peek into.

    Why the Black Box?

    It’s not likely that technology leaders are being purposefully elusive toward business leaders–it’s more likely that they simply are not aligned on what, fundamentally, they need to do to speak the same language.

    The question at hand is, actually, “Are these investments we’re making turning into our desired business outcomes?” But the question technology leaders feel compelled to answer is, “What do your people do all day?”

    These are questions that Agile and DevOps don’t solve–translating Agile or DevOps metrics (also known as DORA metrics) into actual, actionable metrics for business leaders to understand. Although these metrics are tremendously useful for teams, using them to steer the business is a mistake.

    Misaligned in purpose and using different language, business leaders and technology leaders talk past each other as software development costs continue to rise.

    But the cost of software development is too high for business leaders to continue to tolerate this ‘black box’ of ever-rising costs. It’s the fiduciary responsibility of financial leaders to understand the value of software investments and figure out how they are affecting business outcomes–or their jobs could be on the line.

    In case this seems like a distant threat, it isn’t: I saw a headline recently about a prominent automobile executive who lost their job because they weren’t able to gain the necessary insight into, and business outcomes out of, their significant software development costs and that was affecting the competitiveness of their entire company.

    It’s happening. And it’s on both business and technology leaders to break that black box open and start speaking the same language.

    The next question, of course, is, “How?”

    How to Bridge the Communication Gap Between Technology and Business Leaders

    For far too long, business leaders have made do with the activity metrics provided by technology leaders–they’ve tolerated this lack of visibility and found other ‘strategic’ ways to justify the rising software development costs. Perhaps they’ve been questioning how A leads to B–how an increase in funding for software development might result in greater customer retention or app usage, for example–but they have not had a tangible way to prove to themselves, much less to investors, whether this is actually happening.

    In order to understand and quantify software development costs, we have to start demanding more from our technology leaders: Working with them to establish realistic objectives and key results (OKRs), and holding them accountable to business metrics that actually relate to outcomes. That means insisting on business results–and not accepting activity metrics in their stead.

    Stable-Fund Products, Not Projects

    Here’s a hard pill for finance to swallow: When it comes to delivering software-enabled customer experiences, it’s not about optimizing costs. It’s much more nuanced than X dollars in equals X dollars out. Funding by projects might help make the numbers neat, but it doesn’t reflect the reality of the business.

    Project-based funding also tends to keep project leaders in a cost-center mentality: It makes it seem as though all resources are fungible; as if the work and the people performing it are interchangeable.

    One of the goals of Agile is to maximize value for the customer. But when we fund by project, we often stop funding a piece of work–declaring it done–before it actually starts generating value. We have this false end date that doesn’t reflect the actual ending of the value that we funded to create.

    Instead, finance organizations should move to the stable funding of products: Funding an outcome and giving teams the ability to stop, change or extend their work to achieve the business outcomes of the product.

    This is a shift that one prominent UK bank has been going through in recent years–from traditional portfolio management to lean portfolio management, including the shift to stable-funding projects. You can learn more about how NatWest Group (formerly Royal Bank of Scotland) has made this shift in this webinar.

    Visualize the Flow of Value with Flow Metrics

    Finally, we have to stop treating the collection of technology metrics and business metrics as two separate, unrelated efforts. Agile and DevOps are great at accelerating the throughput of your software development … but vanity metrics around how quickly teams are delivering are not enough to justify the growing and material investment in software development.

    Finance leaders have the opportunity to be the Babelfish between technology leaders and business leaders. As mentioned above, one way to encourage clearer communication between teams is to use OKRs to naturally tie software development objectives to tangible business results. The more we practice drawing connections between technology and business metrics–the closer we get to finally breaking open that elusive black box.

    A key part of this is communicating operational metrics differently: Developing a core set of metrics for tracking the flow of business value in software delivery, and providing a mechanism that correlates the investment in flow for each product’s value stream with the business results for that value stream.

    Visualizing the flow of value in a software development process may not be as easy as watching value flow on a manufacturing floor–but that doesn’t mean it’s impossible. Thanks to the vast number of data and visualization tools at our disposal, it is possible to make software products and their value streams just as visible as production lines.

    The Flow Framework, created by Dr. Mik Kersten, addresses the pivotal challenge of breaking open the black box of software development and bridging the gap between business and technology to optimize the flow of business value to customers.

    Break Open the Black Box with Value Stream Management

    The black box of software development is, at its core, a communication issue. It’s on both business and technology leaders to bridge this communication gap. Business leaders need to demand more from technology leaders and insist on translating activity into business metrics. Technology leaders need to practice tying their efforts to tangible business outcomes to help justify their growing expenses.

    Of course, it’s helpful to have tools purpose-built to facilitate this visibility and communication.

  • Building New Value in IT Systems Management

    Building New Value in IT Systems Management

    For decades, companies treated the IT system as a cost center not much different from facilities maintenance. The job of IT was to keep the lights on, ensuring that the technology a company used worked and that the costs of keeping it working were kept under control. Those days are well over. Not only is technology an operational necessity, but everyone now recognizes that the way companies strategically use technology—whether they buy it, rent it or build it—has a direct impact on company performance. This goes beyond efficiency. Technology is key for everything from customer acquisition and engagement to competitive differentiation and advantage.

    As it turns out, however, not every company has been able to rethink and reconfigure its IT system to meet these new demands. That’s not surprising. Not only does this require a mindset shift from the C-suite on down, but it also requires restructuring the IT function to ensure it supports business objectives in a consistent and measurable way.

    While transforming IT from a cost center to a profit center can hardly happen overnight, there are three things that any company can do to turn IT into both a business enabler and a business driver.

    1. Establish Full Transparency Over Your IT System

    A mission-critical IT system needs to have a thorough understanding of what the organization’s IT landscape looks like. Spending money on duplicate applications and unused licenses is a waste. Unknown SaaS applications pose a persistent security risk. Optimizing spend and improving security are table stakes for business-enabling IT.

    The fact that anyone in a company with a credit card can subscribe to a SaaS service is the main challenge. Effectively tracking down all SaaS applications and licenses, along with monitoring SaaS usage, can’t be handled manually. What is needed, as with other tedious tasks, is automation. For example, by integrating a SaaS management platform (SMP) with SSO and expense management solutions, organizations can automate the discovery of SaaS. When the SMP also integrates with the primary SaaS vendors companies rely on (Microsoft, Google, Salesforce, etc.), it allows organizations to keep track of usage.

    The goal here isn’t for a centralized IT function to exercise more control over the software used by organizations. Instead, the goal is leveraging automation strategically to produce the kind of data-driven insights that allow organizations to make better, more informed decisions about the technology it uses.

    2. Promote Engineering Efficiency

    All companies today are software companies. This means that the engineering function needs to be thought of us as part of the overall IT estate. Not only does the IT system need to support engineering as it would any other business function, but the technology created by engineering itself inevitably becomes part of the broader technology landscape managed by IT.

    Mission-critical IT supports engineering efficiency in at least two ways. First, IT should provide the tools and capabilities needed to drive engineering efficiency. For a long time, measuring the efficiency of engineering was either not done at all or was based on meaningless vanity metrics such as lines of code. Recently, DORA metrics have emerged as the scientifically validated method, widely trusted by engineering teams themselves, for tracking efficiency. IT needs to enable easy tracking of these metrics.

    IT should also facilitate the creation and maintenance of a service catalog for engineering teams. Maintenance of this catalog not only promotes the reuse of services, itself a hallmark of engineering efficiency but also makes it possible to quickly mitigate security issues associated with microservices (Log4j being just one example from the last 12 months).

    Beyond efficiency, however, IT can also support in-house software creation by helping engineering teams with value stream management so they can track the business impact of the software they create. While efficiency can have a business impact (realized through time/cost savings), software drives the business forward when it effectively meets customer needs. IT can facilitate these insights by connecting the dots between customer feature requests, product changes and increased adoption (of products and features).

    3. Make Enterprise Architecture Strategic

    An IT function serving as a mission-critical part of the organization cultivates a comprehensive overview of the software landscape and supports engineering efficiency, as suggested, by viewing the development environment as an integral part of the broader technology estate. However, IT system leaders need to do something more to effectively collaborate with the business leaders and drive the business forward.

    Specifically, IT leaders need to provide the business with insight into all the ways technology influences business performance. This means highlighting risks that need to be addressed, uncovering opportunities that should be pursued, and suggesting options for future planning. Thoughtful and strategic enterprise architecture management here is the key.

    A strategic approach to enterprise architecture management means focusing less on governance (who gets to use what) and more on reducing complexity to promote agility and quick decision-making. The first step in this direction is the adoption of a common language that aligns the business and IT. Business capabilities and the creation of business capabilities maps for different departments serve this purpose.

    To make decisions about technology, business leaders need to know what they are working with. Reliable and current data is essential. For this reason, your enterprise architecture management needs to prioritize ongoing data collection and the maintenance of data quality.

    Finally, to make objectives and options transparent, IT needs to be capable of strategically displaying all enterprise architecture data in a way that can be easily understood and acted on.

    Where Do You Start?

    Taming an unruly SaaS landscape, making it possible to build better software and guiding technology strategy are all easier said than done.

    The good news is, ongoing innovation has created solutions that are easier to use, effectively leverage automation and integrate with all relevant systems. In other words, the tools are there. All that’s needed is the commitment to transforming the IT system and developing it into the critical business function it can and should be.

  • Why You Need to Empower Your Developers

    Why You Need to Empower Your Developers

    In recent years, the demand for software developers worldwide has increased. However, in 2020, when the pandemic hit, many CIOs pulled back considerably on their IT spending. This trend has quickly reversed. In 2022, worldwide IT spending is expected to reach $4.4 trillion. Additionally, according to the U.S. Bureau of Labor Statistics, employment in software development is projected to rise by 22%.

    Although organizations across several industries are increasing investment into IT, the supply of technology talent has struggled to keep pace. With the demand for creativity, developers understand their importance and can better dictate their terms — including higher salary compensation and enhanced benefits. Organizations that don’t meet these expectations risk losing coveted talent.

    Remote work continues to be a top priority for many developers, along with great company culture. To create a positive work environment for developers, business leaders must offer a culture that makes tech employees feel a sense of belonging and purpose while working remotely. Many developers also want opportunities for growth and the ability to provide agile and innovative toolsets to help increase productivity and collaboration.

    Let’s discuss some of the ways modern organizations can still empower developers and how business leaders can build a workplace culture that attracts some of the best developers in the business.

    Provide a Supportive Digital Workplace

    Open source usage has increased significantly in the workplace and continues to be a formative resource for developers. The open source community provides many new technologies that developers seek and even require in order to work at the top of their abilities. For example, open source provides some of the most popular programming languages available, including Javascript, C+++, and Python. The presence of multiple languages makes it easier for developers to write code and produce high-quality work efficiently.

    Open source communities help developers build their skills and allow them to customize their technology stacks, as a result allowing them to optimize their productivity and learn different techniques from other open source projects. There is little to no debate that open source is critical to the tech industry and developers. Embracing open source can increase satisfaction and create a better workforce overall.

    Inspire Collaboration Among Software Developers

    In the past, office perks such as free food and collaborative social areas were used to attract new employees and retain current employees. However, in the remote work world, these can no longer remain a priority. Instead, business leaders should prioritize the holistic professional experience that their organization provides its employees. For developers, in particular, building this experience falls on CIOs and IT leaders to ensure they are providing workplace satisfaction, modern technology and flexible processes. To do so, they need to focus on tools that promote productivity and communication.

    Most employees are familiar with the workflow software and collaboration tools that many workplaces provide. Without these offerings, remote workers would feel confused by their workload at the worst and adrift from their co-workers at best. For developers, however, these general collaboration tools won’t cut it, as they are poorly suited for engineering workflows. General collaborative software doesn’t offer the ability for deep customization, nor do they adapt natively to deploy on-prem or in the private cloud. As a result, developers have little to no use for these tools.

    And that lacking sense of utility makes a crucial difference. Developer burnout is at an all-time high and inefficient workflows can keep teams from performing at their best. Or, worse, it can frustrate individual employees, leading to lower retention rates and less attractive offerings for prospective employees. Business leaders looking to offer developers an improved working environment should provide a more integrated, developer-oriented tech stack.

    Empowered Software Developers are Productive Developers

    While the market has changed, for many developers, fragmented tools remain a top productivity challenge. Investing in modern tools and programming languages while also removing tech complexities are the keys to improving productivity and simultaneously creating a better developer experience. This is why organizations must implement open source applications so developers can have complete visibility and control over the entire stack.

    With tool fragmentation causing a number of distractions, developers can become frustrated and spend time responding to every individual incident. This poses the threat of a developer losing their love of the work. By doing away with fragmentation from old technology and selecting relevant solutions, business leaders allow developers to do what they do best: Develop software.

    When organizational leaders empower their developers, they in turn find their product teams are more productive and feel more tied to the organization as well as their customers and each other. Thus, what is best for developers is best for the organization at large.

  • Don’t Let Developer Toil Affect the Business Value of Your Apps

    Don’t Let Developer Toil Affect the Business Value of Your Apps

    It is no surprise that the world today is driven by apps. Organizations all over the world are basing their business goals on their capacity to integrate new apps quickly, and they need to constantly reimagine how they interact with and communicate with their consumers and employees to fulfill changing expectations. To do this effectively, businesses need to operate at a high level of speed and agility. But in many cases, this isn’t happening. The phrase ‘move fast and break things’ could not be further from the truth when it comes to how enterprises approach innovation today. In fact, according to Forrester Consulting, 48% of executives say they haven’t changed their applications in a year or more. One reason for this may be that organizations are dealing with a high amount of developer toil and technical debt. Developer toil is the repetitive, predictable, constant stream of tasks that support the creation of software and how it’s run, but don’t actually affect the features in the software that people use. Or in other words “any activities that don’t directly create business value.”

    Typically, developers spend too much time on processes that could be automated or even eliminated. Ironically, this toil builds up over time as their organizations prioritize shipping features rather than addressing these problems in their overall software process. While developers can change their software quickly at first, just like debt in real life, if this toil and tech debt are neglected, it takes over and consumes the organization’s ability to grow. This results in long, infrequent release cycles. A feature that seemed simple and once took just 15 minutes now takes weeks, even months, to get in front of users.

    Conducting a Developer Toil Audit

    Anything you can do to speed up your software release cycle will improve the quality, resilience and business value of your software. Continuously addressing developer toil provides a significant boost to your organization’s software capabilities. For one of VMware’s customers, addressing developer toil reduced the time taken to introduce a new feature significantly–from 400 hours to just 24 by doing a developer toil audit and addressing key factors contributing to this technical debt.

    So how can organizations overcome developer toil? One way to do this is through a developer toil audit. This is a systematic process for finding, valuing and prioritizing fixing waste in your software process. First, the process finds wasted time and process debt in how you build and release software. Second, the process then helps you justify delaying working on features in favor of eliminating and automating your software creation process. The result is speeding up your software release cycle. The more frequently you can change and release software, even with just small changes, the more opportunities you must learn what works and put those features in front of customers, employees, and other users.

    The process of conducting a Developer Toil audit includes:

    1. Asking the right questions: A great way to uncover developer toil is to ask about people’s struggles, frustrations, and even boredom in the form of a tailored question set. We advise using questions that are specific to your organization in addition to general questions like “how long does it take to perform a full build?” For example, highly regulated organizations should ask about tasks involving compliance. Once the survey is completed, you can do some quick analysis to locate and prioritize developer toil.

    2. Combine into usable metrics: You now have a single metric for each type of developer toil. And by combining all the questions you create a single metric to track overall developer toil. Extra points for creating dashboards with visualizations! As with all such aggregate metrics, these are more directional than exacting—their job is to point you to problems that should be solved.

    3. Link these metrics to business value: A simple ranking of developer toil is better than nothing. However, before deciding which developer toil to address, we recommend linking each type of developer toil to the business value created by fixing the toil. Put briefly, the value you get will be related to the time saved, and, thus, the ability to ship more features in the future.

    The less time developers spend on toil, the more time they can spend on tasks that directly benefit the business. Another way of looking at this is plain old productivity: Developers can now do more with the same amount of time.

    Another important outcome is increased staff morale. The less time developers spend on boring, repetitive work—toil—the happier they’ll be. Stronger employee hiring ability and retention are (or should be) a strategic imperative for any organization that depends on software. Morale also increases software quality: happier people make better software.

    Slow Down to Speed Up

    With the survey results in hand, you should have a pretty good idea of which developer toil to fix. The last step is to do the usual product management prioritizing to weight fixing these items with other tasks you could do. These are strategic decisions that product managers need to make. To fix toil, you need to stop shipping features to free up time. You need to slow down the business. At the start, before product managers have been empowered to make these kinds of decisions, higher-level management should be involved to calibrate how much slow down you’re willing to put up with for future agility. The calibration you’re doing is this: If I fix one item of toil and ship just one feature (instead of two) this release cycle, then in the next release cycle I can ship four features. Humans are not great at that kind of bird-in-the-hand versus birds-in-the-bush thinking, so you’ll need to figure out what works in your organization.

    Through our experience as consultants working with product teams in different industries, we’ve used Developer Toil Audits to focus and motivate product managers and developers to fix toil and speed up their release cycles and, thus, improve how their organizations build software. This directly improves the business by adding more capabilities and capacity to deliver more features, even in the short term. Each time you fix any technical debt, you gain more capacity and capabilities to create new features and improve your software. This is how you use a modern software culture to increase business agility. Or, put more simply: Less developer toil leads to better software, and better software leads to better business.

  • Building a Culture of Full-Service Ownership

    Building a Culture of Full-Service Ownership

    In recent years, our reliance on software has exploded. It has been more than a decade since Marc Andreessen made his prophetic observation that “Software is eating the world.” Today, nearly every interaction is mediated through a screen and backed by software. With the massive shift to remote and digital interactions due to the pandemic, enterprises that were already in the process of acclimating to a digital-first future had to quickly pick up the pace to survive today’s increasingly complex tech landscape, putting an inordinate strain on IT teams.

    Developers obviously deliver the most crucial component of the software development life cycle; however, ownership of their developed services and applications is no less important. The idea of ‘full-service ownership’ means taking responsibility for supporting the software and services you deliver at every stage of their life cycle. Excellence in continuous delivery and innovation on them is about speed and velocity while maintaining quality.

    At its core, service ownership is about connecting humans to technologies and services and understanding how they map to critical business outcomes. Achieving this level of ownership requires an understanding of what and who delivers critical business services. A clear understanding of what the boundaries and dependencies are of a given service along with what value it delivers is the starting point. And once it’s in production, a clear definition of who is responsible for it at any given time and what its impact is if it isn’t running optimally or, worst case, fails altogether. Empowering developers with this information brings DevOps teams much closer to their customers, the business and the value they create which, in turn, leads to better application design and development.

    Building a culture around a full-service ownership model keeps developers closely tied to the applications they build and, therefore, closer to the value they deliver. Within the organization, this type of ownership breaks down long-established centralized and siloed teams into cross-functional full-service teams. For enterprises with deeply ingrained processes, this may be easier said than done—but the outcome is well worth the investment. Following these guidelines will allow even legacy IT teams to create a culture of full-service ownership.

    Get Developers on Board for Clear Ownership

    With the pressure to innovate already weighing heavily on software teams, you may be wary about the burden more responsibility might cause. However, the end goal is mutually beneficial: Create clear ownership at real-time speeds. Not only does this allow engineering teams to ship features more quickly but it also empowers them to stay engaged and connected when something goes wrong—reducing hand-offs and mean-time-to-resolve (MTTR). In other words, no panic, no uncertainty and minimal customer impact.

    Decentralize Team Structures

    Reworking team structures and technology for full-service ownership may seem overwhelming, but it doesn’t need to be. Traditional function-based silos are a feature of many legacy IT departments. Provisioning infrastructure and resources, developing code, quality assurance, owning the customer relationship, supporting software in production and so on are too often delegated across segmented, centralized teams. Allowing decentralized teams to own the entire life cycle of their services affords them the ability to add value in new ways. The key is to start small and implement changes gradually.

    Implement a Strategic Rollout

    When beginning the process of full-service ownership adoption, pointing to small wins will gain the support of key organizational stakeholders who can then facilitate larger-scale change. Start with non-critical production systems or greenfield applications. Agile methodologies will also be useful in adapting to dynamic development needs and unpredictable events. Remember to be flexible with the timeline in the initial phases and be patient with your team—mastering the life cycle takes time!

    Keeping long-term goals in mind, your teams should consider not only “build and deploy” but also “run.” Craft your tech stack in a way that comprehensively shows the entire infrastructure, leverages machine learning to reduce false alarms, and uses automation to conserve the manpower and resources involved in going on-call to support services in production.

    Leverage the Right Tools and Technology

    A key requirement here is building trust and an accurate system for identifying who is responsible when something goes wrong. Adjusting to a DevOps approach will fail if developers receive false alarms, too many interruptions or are constantly escalated to repetitive fixes. Building and executing on a strong service directory and supporting their preferred schedules is critical. As is ensuring when they are needed, you contact them via the channels they expect. Failing to do this can result in minutes and hours lost and worse yet, frustrated and burnt-out developers. Recent data shows that developers who get called in to support services too often or incorrectly are more likely to leave their jobs, and that’s only going to stall your progress further.

    With an accurate service directory in place, it is important to remove the false alarms. Teams must make sure “alerts and alarms” are correlated and an incident is confirmed before you interrupt your valuable time-strapped developers. This involves the right observability and monitoring tools but must also involve strong centralized machine learning augmentation to reduce noise and false alarms. As teams mature, automation plays a critical role here. First, by cutting out the common diagnosis and investigation steps that can burn minutes or even hours. Then, reducing escalations to developers (by over 50%) by putting common resolutions in the hands of first responders or even customer service teams. These collections of technologies are commonly referred to as AIOps and as digital business scales, they become a foundational requirement.

    Cross-Functional Teams are Table Stakes

    Today’s digital ecosystem is more advanced and more complex than ever before. Maintaining quality digital services is not just a competitive differentiator, it’s the backbone of your corporate reputation. Recent high-profile outages and cyberattacks have catapulted the fragility of the software supply chain into the international spotlight, meaning the margin for error is slim. Resolving issues proactively and in real-time before they impact customers is critical.

    Transparent accountability and decentralized ownership ensure that teams have a deeper knowledge of issues when trouble strikes and can take swift action. Revealing to technical teams and developers the direct impact they have on business outcomes enables high-quality, innovative code that delivers on its promises.

  • 4 Reasons Software Developers Need a Bill of Materials

    4 Reasons Software Developers Need a Bill of Materials

    The recent Log4j/Log4Shell vulnerability was a wake-up call that threats aren’t going to wait until the industry gets up to speed on software supply chain security. While the Log4j open source component vulnerability caught us all off guard, it did highlight the need for software vendors to be more proactive in disclosing the composition of their products. It’s become painfully clear that analyzing open source libraries and third-party code should become standard practice to understand their makeup and the potential risks they introduce in commercial software applications.

    Unfortunately, relying on open source projects to provide visibility into security vulnerabilities isn’t going to work. In most cases, these projects are grossly underfunded. At the time of the Heartbleed vulnerability, for example, OpenSSL received very little funding. Yet, open source usage increases each year. It’s imperative that commercial software organizations contribute to open source projects they use and share responsibility for the security of these products.

    In addition to open source libraries, most vendors source third-party code from contractors or for-hire development organizations. Without documentation of the composition of the code developed by third parties, it’s impossible for commercial software vendors to proactively identify and manage risk before it’s implemented in their development cycle.

    Using Software Composition Analysis (SCA) Tools

    Where a software bill of materials (SBOM) is not provided, the onus should be on commercial software vendors to create one. This can be accomplished using software composition analysis (SCA) tools that analyze binaries and do not require access to source code. They can generate SBOMs to identify open source or third-party components in use, as well as track and manage security risks in the software supply chain.

    Creating a comprehensive SBOM provides other benefits, since most open-source projects and contract developers are only likely to publish direct dependencies (if at all) and not look further into the dependencies of dependencies, and so on.

    Four Benefits to Consider

    Meanwhile, commercial software vendors can expect their customers to soon start demanding SBOMs since this will be a requirement under the cybersecurity executive order. Given this evolving landscape, commercial software vendors should consider the following four benefits for proactively implementing an SBOM program to govern the products they build:

    • Gain visibility into software supply chain risk to identify, reduce and remove vulnerabilities in reused software (i.e. open source and third-party). SBOMs provide data for business decisions on the software you have reused and purchased. It’s better to discover these risks before your customers do.
    • Preemptive supply chain qualification to ensure compliance with future requirements. SBOMs will soon be a requirement for doing business with the U.S. federal government. Suppliers that meet the SBOM requirements during procurement will be given preferential treatment.
    • Improved security and downstream benefits that come with risk management and mitigation. Avoiding, detecting and remediating security risks before they become embedded in a product pays huge dividends during development and deployment. You’ll be better prepared for the next security event.
    • Common understanding of software assets that comes with a standardized SBOM shared amongst developers, suppliers and open source projects. An SBOM becomes a way to communicate software contents and dependencies within and outside an organization.

    Taking a proactive approach to the software supply chain by creating SBOMs and demanding the same from suppliers will enable commercial developers to be better prepared for the next Log4j/Log4Shell vulnerability. It will also allow them to improve the software supply chain security posture of their end customers.

  • Securing Software with Intelligent Pipelines

    Securing Software with Intelligent Pipelines

    One of the biggest cybersecurity risks involves vulnerabilities in the application layer. After all, the best firewall is useless if the web application itself is vulnerable. Many companies have worked to mitigate these risks by investing in their AppSec programs. According to a recent whitepaper written by ESG (commissioned by Synopsys), 71% of companies surveyed now use AppSec tools for more than half of their software projects. Notably, over two-thirds of companies already use 11 or more automated application security testing (AST) tools, such as SAST, DAST, IAST, fuzz testing and container scanning solutions.

    This is due, at least in part, to the fact that tool manufacturers have now made their products “DevOps-ready” and support suitable integrations with CI/CD pipelines. This makes it tempting to simply let AppSec scanners run in the pipelines, but that can introduce other problems.

    Problems with AppSec in CI/CD Pipelines

    Too many results: Developers can be inundated with findings, yet only a small percentage are likely to pose such a high risk that they need to be fixed immediately. But the prioritization guidelines are often formulated in separate documents and are ambiguous.

    Development pipelines are slowed down: Build pipelines often run at frequent intervals; every second to every minute. Scans with AppSec tools may take several minutes or even hours.

    Manual AppSec activities are left out: Not all AppSec activities can be automated, such as architecture risk analyses, threat models and penetration tests. Nevertheless, these are an essential part of the AppSec strategy.

    Intelligent pipelines (i.e., intelligent, purpose-optimized automation and orchestration of the various AppSec tools and activities) are ideal for overcoming this challenge. Combined with the consolidation of scan results, a new category of solutions has emerged here, which Gartner dubbed application security orchestration and correlation, or ASOC for short, in 2019.

    How Pipelines Become Intelligent

    The “intelligence” lies in deciding which tools need to run at what time and what to do based on the results. So instead of scanning the entire codebase with AppSec tools at every commit, it dynamically decides which scanner needs to run and to what extent. This decision can take into account various parameters such as the scope of the actual code change, the risk profile of the application or the development stage of the software.

    The risk profile of the application should also be considered. Web applications that are accessible from the internet and process sensitive data pose a greater security risk than an internal tool for generating documentation. Such risk profiles usually emerge from prior architectural risk analyses and threat models.

    Furthermore, the scope of AppSec testing should be appropriate to the development stage of the application. Individual commits of a feature branch should be checked mainly by static code analysis for passwords and API tokens contained in the code and compliance with coding guidelines, such as SEI CERT, to support rapid development. Later on, during the merge request into the main branch, more extensive scans should be added, including deeper data flow analyses, which then detect cross-site scripting or SQL injection attacks. A longer runtime can be accepted here since such merge requests usually have to be approved according to the dual control principle.

    Code Security Policies

    The core of intelligent pipelines lies in individual guidelines or policies. These define when specific AppSec activities are executed. Additionally, these policies describe how to proceed with the combined results (e.g., whether the code may be integrated into the master branch or the web application may go live).

    The policies are described in a configuration file; this is policy–as-code. Just as with other as-code methods, this enables or improves uniqueness, reproducibility and automation. Simple policies can be created according to the “If this, then that” principle. For example, a software composition analysis (SCA) scan can be triggered when either the project file changes (package.json, go.mod, pom.xml, etc.) or new files or directories are added, but not when only existing source code files are changed.

    Conclusion

    Whether with or without orchestration tools, anyone who wants to make their software more secure must inevitably consider which AppSec activities make sense at what time. In other words: The right scan at the right time. Furthermore, it should be determined how to proceed with the results before ordering or even automating a scan. To do this, you need to understand what the greatest risks are and how to protect the software from those risks.

    Intelligent pipelines help to implement and, above all, automate an AppSec strategy. A corresponding strategy is a prerequisite. But even (or especially) without a clear strategy, it makes sense to look at the architecture of an intelligent pipeline, as this both encourages collaboration between development, AppSec and DevOps teams and raises the right questions that lead to a successful AppSec strategy.

  • Using Open Source to Secure Software Supply Chains

    Using Open Source to Secure Software Supply Chains

    Recently, there’s been a lot of attention paid to software supply chain security. In particular, here’s a quote from the May 2021 presidential executive order on improving the nation’s cybersecurity: “The Federal government must … advance toward zero trust architecture; accelerate movement to secure cloud services, including … platform as a service (PaaS).”

    There are two parts necessary to create a truly trusted software supply chain; securing the non-technical areas and securing the technical areas.

    Non-technical aspects of any secure software supply chain involve having individuals or teams focused on security and audit compliance. Internal company policies that act as a regulatory system and set standards for developers are a must, as are efforts to enforce compliance with security best practices. While this can bode well for large organizations, small software engineering teams and startups do not have the bandwidth, budget or culture to make this a reality.

    Envisioning Robust Security Best Practices

    Tools that are open source, governed strictly and enable automation of secure build and deployment are the components that form the technical aspects of the solution. Engineering teams must find a way to envision robust security best practices and find a way to apply them without unduly affecting the developer workflow. This is a founding principle of the DevSecOps efforts within the larger community of software development professionals.

    Ergo, advocating a technology stack that:

    • Is composed of open components
    • Provides automated builds
    • Reduces the dependency on developers
    • Allows security operators to extend control
    • and is actively supported by the community is an ideal choice upon which to build a platform for use by development teams.

    The focus of this article is to describe a robust stack that will sit above compute resources and power applications. The stack will consist of fully customizable open source components that put reliability and security at the forefront.

    Simplify Operations

    Application source code is the single source of truth when working with this stack. The use of Git, a popular version control system that is free and open source will allow developers to work with a source code management system from which all the downstream steps will be triggered.

    Kubernetes is an open source container orchestration tool developed under the auspices of the Cloud Native Computing Foundation (CNCF). It serves as an abstraction over the infrastructure and compute layers to power the system from underneath. Kubernetes has managed to gather a large community following—second only to the Linux OS itself. Using Kubernetes will bring homogeneity above the infrastructure layer and will simplify further operations.

    Cloud Foundry is an open-source PaaS tool. The use of Kubernetes introduces complexity that introduces a bit of overhead for developers who work with the platform. Cloud Foundry provides a countermeasure that simplifies the developer experience and greatly eases the pain points commonly associated with Kubernetes adoption. The Cloud Foundry platform does the job of deploying all the applications to the Kubernetes infrastructure. The cf push command triggers the export of a container image from the application source code.

    When building these containers to deploy to the runtime within Kubernetes, Cloud Foundry uses Paketo buildpacks internally. Paketo buildpacks are an implementation of the cloud-native buildpacks specification which aims to provide a unified means to generate OCI-compatible container images for all languages and frameworks that are commonly used to build applications. Paketo buildpacks are also open source and fully customizable according to the needs of software development teams.

    A Trusted Architecture

    Together, this platform, built on open tools, supported actively by the community and facilitating the exact needs of software developers helps promote a trusted architecture upon which to stage applications. Using these tools would enable an organization to build a software development ecosystem that satisfies all the requirements implied and enumerated in the various recommendations for improving software security.

  • Report: Software Quality Issues Costing U.S. Companies $2.08 Trillion

    Report: Software Quality Issues Costing U.S. Companies $2.08 Trillion

    A report published today by the Consortium for Information & Software Quality (CISQ) estimates poor software quality collectively cost companies in the U.S. an estimated $2.08 trillion in 2020.

    The report, co-sponsored by software quality and security tools provider Synopsys, attempts to calculate, at least theoretically, the cost of known software failures. The report identifies operational software failures, estimated to be $1.56 trillion, as the single largest cost. The next-largest cost contributor is unsuccessful development projects, which totals $260 billion, an estimated 46% increase since 2018, according to the report. The project failure rate remained steady, at about 19%, for over a decade, according to the report.

    Finally, legacy systems issues contributed $520 billion to costs in 2020, a significant decrease from the $635 billion these issues cost in 2018.

    That estimate does not include technical debt, which CISQ estimates at another $1.13 trillion. Technical debt costs have been growing at a rate of 14% per year since 2018, the report noted.

    Joe Jarzombek, director for government and critical infrastructure programs for Synopsys, said it’s apparent there’s not enough focus on quality assurance within software supply chains. However, as organizations begin to realize how dependent their digital business transformation initiatives are on quality software, business executives will be forced to shift focus.

    Unfortunately, most business executives today still don’t understand or appreciate how software is constructed. Most organizations don’t even have a bill of materials describing the components that make up an application, noted Jarzombek.

    That issue becomes especially problematic in the event of a security breach, because too many organizations can’t draw a correlation between vulnerabilities and the impact on specific components of an application, added Jarzombek. At the same time, however, cybercriminals are becoming more efficient at scanning applications to discover and exploit those vulnerabilities, said Jarzombek.

    Many organizations also don’t appreciate how adept cybercriminals have become at exploiting what are perceived as low-level vulnerabilities to distribute malware laterally across an application, said Jarzombek. There is a tendency to patch only the most critical vulnerabilities within an application, while the rest are left to future updates. When fixing low-level vulnerabilities is not prioritized, these vulnerabilities may remain over multiple release cycles.

    Ultimately, Jarzombek said, rather than treating security separately from the application development process, it should be addressed as part of a comprehensive approach to software quality assurance. The immediate challenge, however, is that not enough developers have access to the tools they need to address these ongoing cybersecurity issues.

    The impact of poor software quality on individual companies will vary widely, but poor software quality is itself a universal issue. Most software development teams are not deliberately ignoring these issues; instead, the processes currently employed to build software are not especially robust, mature or, for that matter, automated, leaving organizations vulnerable.