devopsy.kinsta.cloud

Category: Continuous Delivery

Continuous delivery is a software engineering discipline that allows organizations to consistently develop and release quality software at a rapid pace. It is achieved by automating the software development lifecycle, enabling tasks such as development, testing and release to occur concurrently and continuously.

  • Building a Culture of Full-Service Ownership

    Building a Culture of Full-Service Ownership

    In recent years, our reliance on software has exploded. It has been more than a decade since Marc Andreessen made his prophetic observation that “Software is eating the world.” Today, nearly every interaction is mediated through a screen and backed by software. With the massive shift to remote and digital interactions due to the pandemic, enterprises that were already in the process of acclimating to a digital-first future had to quickly pick up the pace to survive today’s increasingly complex tech landscape, putting an inordinate strain on IT teams.

    Developers obviously deliver the most crucial component of the software development life cycle; however, ownership of their developed services and applications is no less important. The idea of ‘full-service ownership’ means taking responsibility for supporting the software and services you deliver at every stage of their life cycle. Excellence in continuous delivery and innovation on them is about speed and velocity while maintaining quality.

    At its core, service ownership is about connecting humans to technologies and services and understanding how they map to critical business outcomes. Achieving this level of ownership requires an understanding of what and who delivers critical business services. A clear understanding of what the boundaries and dependencies are of a given service along with what value it delivers is the starting point. And once it’s in production, a clear definition of who is responsible for it at any given time and what its impact is if it isn’t running optimally or, worst case, fails altogether. Empowering developers with this information brings DevOps teams much closer to their customers, the business and the value they create which, in turn, leads to better application design and development.

    Building a culture around a full-service ownership model keeps developers closely tied to the applications they build and, therefore, closer to the value they deliver. Within the organization, this type of ownership breaks down long-established centralized and siloed teams into cross-functional full-service teams. For enterprises with deeply ingrained processes, this may be easier said than done—but the outcome is well worth the investment. Following these guidelines will allow even legacy IT teams to create a culture of full-service ownership.

    Get Developers on Board for Clear Ownership

    With the pressure to innovate already weighing heavily on software teams, you may be wary about the burden more responsibility might cause. However, the end goal is mutually beneficial: Create clear ownership at real-time speeds. Not only does this allow engineering teams to ship features more quickly but it also empowers them to stay engaged and connected when something goes wrong—reducing hand-offs and mean-time-to-resolve (MTTR). In other words, no panic, no uncertainty and minimal customer impact.

    Decentralize Team Structures

    Reworking team structures and technology for full-service ownership may seem overwhelming, but it doesn’t need to be. Traditional function-based silos are a feature of many legacy IT departments. Provisioning infrastructure and resources, developing code, quality assurance, owning the customer relationship, supporting software in production and so on are too often delegated across segmented, centralized teams. Allowing decentralized teams to own the entire life cycle of their services affords them the ability to add value in new ways. The key is to start small and implement changes gradually.

    Implement a Strategic Rollout

    When beginning the process of full-service ownership adoption, pointing to small wins will gain the support of key organizational stakeholders who can then facilitate larger-scale change. Start with non-critical production systems or greenfield applications. Agile methodologies will also be useful in adapting to dynamic development needs and unpredictable events. Remember to be flexible with the timeline in the initial phases and be patient with your team—mastering the life cycle takes time!

    Keeping long-term goals in mind, your teams should consider not only “build and deploy” but also “run.” Craft your tech stack in a way that comprehensively shows the entire infrastructure, leverages machine learning to reduce false alarms, and uses automation to conserve the manpower and resources involved in going on-call to support services in production.

    Leverage the Right Tools and Technology

    A key requirement here is building trust and an accurate system for identifying who is responsible when something goes wrong. Adjusting to a DevOps approach will fail if developers receive false alarms, too many interruptions or are constantly escalated to repetitive fixes. Building and executing on a strong service directory and supporting their preferred schedules is critical. As is ensuring when they are needed, you contact them via the channels they expect. Failing to do this can result in minutes and hours lost and worse yet, frustrated and burnt-out developers. Recent data shows that developers who get called in to support services too often or incorrectly are more likely to leave their jobs, and that’s only going to stall your progress further.

    With an accurate service directory in place, it is important to remove the false alarms. Teams must make sure “alerts and alarms” are correlated and an incident is confirmed before you interrupt your valuable time-strapped developers. This involves the right observability and monitoring tools but must also involve strong centralized machine learning augmentation to reduce noise and false alarms. As teams mature, automation plays a critical role here. First, by cutting out the common diagnosis and investigation steps that can burn minutes or even hours. Then, reducing escalations to developers (by over 50%) by putting common resolutions in the hands of first responders or even customer service teams. These collections of technologies are commonly referred to as AIOps and as digital business scales, they become a foundational requirement.

    Cross-Functional Teams are Table Stakes

    Today’s digital ecosystem is more advanced and more complex than ever before. Maintaining quality digital services is not just a competitive differentiator, it’s the backbone of your corporate reputation. Recent high-profile outages and cyberattacks have catapulted the fragility of the software supply chain into the international spotlight, meaning the margin for error is slim. Resolving issues proactively and in real-time before they impact customers is critical.

    Transparent accountability and decentralized ownership ensure that teams have a deeper knowledge of issues when trouble strikes and can take swift action. Revealing to technical teams and developers the direct impact they have on business outcomes enables high-quality, innovative code that delivers on its promises.

  • Secure Software Summit: Behold the SBOM

    Secure Software Summit: Behold the SBOM

    With supply chain security becoming more of a focus, the SBOM is now viewed as a critical element in shoring up supply chain security. SBOM stands for software bill of materials. At a very elementary level, an SBOM is a list of ingredients. Think about how every food product in the supermarket lists ingredients so, for example, people who are allergic to nuts can avoid nuts. An SBOM does the same thing for software applications. 

    If you want to do business with the U.S. government, SBOMs are no longer optional. Executive order 14028 requires an SBOM for all U.S. federal software procurements. Why the push for SBOMs now? Software companies generally do not list all the components included in the software they produce. Knowing what components are included in applications is essential for proactive vulnerability management. That said, the government mandate for SBOMs only requires a minimal amount of included information—supplier name, the component name and version, dependency relationships, time stamps for versions or modification and information about who authored the SBOM itself. 

    What to Look For in an SBOM

    This minimum required data yields minimal benefit for cybersecurity risk reduction and vulnerability management. SBOMs can actually include many other types of information. What information should you include in SBOMs for software applications produced, purchased or consumed by your organization? Here is a list. 

    Authenticity —Authenticity asks and verifies that someone actually created a listed component. Ideally, this requires a digital signature of some sort. Authenticity can be particularly important in the case of nested components where the supplier of an application includes multiple other components, as is common with JavaScript and Java libraries used for common functionalities. 

    Integrity —This addresses whether a component has been tampered with or modified without authorization. An SBOM should include some measure of integrity. Equally important is proving the integrity of the SBOM itself. Just as CheckSum can tell you whether software you are loading matches what the source of that software indicates, integrity checks should allow you to verify that components on an SBOM (and the SBOM itself) are exactly what you are expecting. 

    Services —An inventory of services called by an application is critical. Modern software does not live in a vacuum. It calls out to the Internet to perform all sorts of tasks and communication with other systems. Those services that extend beyond your application are part of the inventory because they can be an indication of vulnerability or indicate a larger problem.

    Vulnerability Remediation Status —Development teams are constantly balancing the need to remediate vulnerabilities against the risk that remediation will break the application. Some open source projects, for example, have a history of not backporting security fixes. The latest version of the project may break applications. Organizations are forced to choose between breaking an application or taking the risk of introducing a high-severity vulnerability that might be exploited. Responsible organizations that opt to keep the older version and not break their software will backport their own fixes. An SBOM should describe these variations and any other vulnerability remediation activity. 

    Inventory of component security advisories —An SBOM should describe security advisories pending against all components. This is likely to be a dynamic list that you will need to automatically update using one of the accepted vulnerability advisory formats (CSAF and CycloneDX, to name two). In fact, this is one of the strengths of SBOMs—they are designed for automation and to act programmatically. 

    Completeness —It is not always possible to ship complete SBOMs, either for technical or intellectual property reasons. An SBOM should indicate whether it is complete and whether it contains a comprehensive inventory of dependencies and services. 

    Pedigree —This information describes the DNA of the SBOM and the original makeup and changelog of each particular component. An SBOM should describe the original component, exactly what changes were made, who committed the changes and why, what the differences were between different versions and what security issues were fixed with each variation and version. 

    Provenance —This relates to the origin of a component or service. It can describe a repository that the component was downloaded from, a country of origin, the name of a software manufacturer or open source project or other information indicating the component or service’s source of origin. 

    License type —License information is relevant for compliance and to ensure that your organization is not consuming or producing software in a manner that violates licensing terms. Moreso than for security and vulnerability, this information is for risk management, compliance enforcement and auditing. 

    Customizing SBOMs to Meet Your Needs

    The SBOM is a flexible and highly adaptable approach that can be easily customized to fit specific use cases. You can design SBOMs to meet the specific needs of your use case, either as a software provider or consumer. You also may need to modify your SBOM fields and data design as your needs evolve. Fortunately, it is rapidly becoming easier to automatically generate SBOMs using software security tools such as software composition analysis (SCA) and interactive application security testing (IAST) solutions. SBOMs can also be created or read automatically at other points during the build life cycle along the continuous integration/continuous deployment (CI/CD) process. Just as you would not buy a box of cereal that did not list the ingredients, in the future you will not want to consume software that does not have a detailed SBOM. What you don’t know about the software you run can hurt you, and SBOMs will be a crucial tool in providing greater transparency and observability into the software supply chain. 

    About Secure Software Summit
    The Secure Software Summit brought together the world’s leading innovators, practitioners and academics of secure software development to share and teach the latest methods and breakthroughs on secure coding and deployment practices. If you are about developing, releasing and securing software, delivering new features fast and building things right from the start, click on the link below to get access to all of the sessions from this summit.

    https://go.shiftleft.io/secure-software-summit-2022-replay

  • Your DevOps Teams Can Do Better. Here’s How.

    Your DevOps Teams Can Do Better. Here’s How.

    As more organizations focus on digital transformation as the centerpiece of their growth strategy, software developers and IT teams are under heightened pressure to perform—quickly and efficiently.

    That begs two questions: Can we do better? And if so, how can we do better?

    The answers can be elusive, especially for DevOps teams that have over-relied on processes and models that, while previously successful, may not be as relevant for today’s projects. Of course, the point about relevance has been further magnified during the pandemic, when sudden shifts to remote working left many teams scrambling to ensure they had the capacity and infrastructure to support massive work-from-home cohorts.

    Software development and IT operations support teams that are winning by doing better in this new era are truly embracing Agile. And that means taking to heart the definition of “Agile,” which is a mindset of values and principles to continually evolve your way of working (WoW) to ensure your productivity and value are tailored to fit the tasks at hand.

    Conversely, teams that have relied on their go-to framework or approach for every assignment are often missing the boat with respect to efficiencies, effectiveness and even ROI. This is because they aren’t assessing whether their proven WoW is useful for their current projects.

    That can lead to underwhelming results. A report from Digital.ai revealed that while over 90% of software development teams take an Agile approach to their work, a whopping 96% of their digital transformation projects have failed or been delayed. Clearly, that suggests Agile has worked for them in the past, but the nature of digital transformation projects is calling for teams to adapt to new or hybrid approaches that may differ from their go-to WoW.

    Challenging? Yes. Doable? Also Yes

    While there is no quick fix to attain true flexibility amid complexity, the journey begins with the recognition that, yes, you can do better. And to do better—to become an adaptive, lean, responsive and learning organization under any circumstances—there are a number of steps to take to ensure your journey to Agile freedom is successful.

    Here are five that I believe must be considered for DevOps teams to become better and more effective:

    • Consider all options. Agile has been practiced for more than two decades, and literally hundreds of frameworks and approaches have been developed and implemented. If you examine your WoW in a context-driven, pragmatic manner you are more likely to select the best-fitting technique given the situation.
    • Do your homework. Your team leaders may not be familiar with the breadth of the Agile, traditional and hybrid frameworks at their disposal. But there’s help available. Seek out online courses and certifications designed to help organizations and individuals explore and navigate the multitude of Agile frameworks that they may currently be overlooking.
    • Focus on value. It’s great if your gut tells you that your Agile framework is reaping real results, but can you prove it? If your senior leadership has invested in Agile training, coaching and certifications, they’ll demand to see some tangible returns. With well-chosen Agile approaches that match the work required, you will be able to deliver the proof that Agile is driving measurable value that matters. Even further, learning which metrics to use in different situations can help measure this value.
    • Sidestep bottlenecks. By choosing the best WoW for the project at hand, you are more likely to eliminate the wasteful or inefficient processes that can add hours or days to any assignment. Understanding how to unclog these pipes is critical in improving the flow of value to your customers.
    • Motivate with morale. If your team is frustrated and morale is low, productivity, the quality of the work and your retention rate will be, too. But, if you’re able to do Agile well, this can help your teams be happier, more productive and more successful in accomplishing project goals. Ultimately, that means a more joyful customer base, too.

    Conclusion

    In the long run, the framework or approach you choose isn’t as important as your willingness to be open-minded in applying Agile frameworks to projects on a case-by-case basis. As you become more comfortable with new or different approaches, you will promote continuous improvement through carefully considered experimentation.

    That means you’ll become a learning organization, one that is open to change and evolution. With that mindset, your development and IT ops team will do better—guaranteed. Because true agility comes from freedom, not frameworks.

  • Survey Sees App Testing More Widely Distributed

    Survey Sees App Testing More Widely Distributed

    A survey of 900 application testing professionals conducted by PractiTest, a provider of a platform for managing testing tools, suggested the number of organizations that have dedicated testing teams is declining at a steady rate.

    The survey found a 15% decrease in organizations with dedicated testing teams that have six or more members. At the same time, however, the survey found only 18% of respondents reported that all testing is done by individuals that are not part of a dedicated testing team.

    Joel Montvelisky, test specialist and chief solution architect at PractiTest, said that shift indicates that more organizations are steadily incorporating testing within DevOps workflows. In fact, more than 60% of respondents said DevOps and Agile processes led to teams being more organized and stable, with two-thirds (66%) reporting that development teams were releasing features more frequently with more functionality.

    However, only 38% of respondents said they were active participants in a continuous integration/continuous delivery (CI/CD) workflow. A full 90% said they needed to coach or train other members within the team and just over half (54%) reported that estimations of the time required for development and testing were accurate.

    Two-thirds also said collaboration with developers is also improved when applying DevOps processes, with more than half reporting fewer serious bugs made it into production. More than half also said they monitor software in production and deployment environments. The number of testers who defined themselves as engineers has gone up by 14% year-over-year. However, only 44% report that automation has replaced 50% or more of their manual testing efforts.

    A full 95% or more of survey respondents identified application programming interface (API) testing, Agile methodologies, functional testing and scripting as high on their list of priorities. In contrast, big data testing, internet of things (IoT) testing and machine learning/artificial intelligence (AI) testing were cited as important or very important by only 60% of respondents.
    Finally, about a third of survey respondents noted that non-testers are now involved in testing activities.

    Montvelisky said the PractiTest survey makes it clear that while developers are a little more involved in testing than they were a few years ago, organizations still relied on testing specialists that are becoming more integrated into DevOps workflows. This is especially important as more organizations launch digital business transformation initiatives that require them to build software using the same types of DevOps processes that independent software vendors (ISVs) employ.

    It’s still too early to say just how far testing might shift left toward developers, but as testing becomes more automated—thanks, in part, to increased reliance on AI—the types of tests that will be conducted by a dedicated testing team will evolve. In fact, it’s unlikely automation will eliminate the need for those teams any time soon. Rather, testers will be augmented by AI technologies that will help them to keep pace with the accelerated rate of application development enabled by wider adoption of DevOps best practices within organizations both large and small.

  • Secure Software Summit: Applying Chaos Engineering to Software Security

    Secure Software Summit: Applying Chaos Engineering to Software Security

    Today’s software systems are, essentially, controlled chaos—and lightly controlled chaos, at that. This makes it exceptionally challenging to model the behavior of those systems. Our systems are quickly becoming larger and larger, with more and more moving parts. It is not uncommon for enterprises to have over 1,000 microservices and millions of containers running thousands of applications containing many thousands of open source libraries. 

    Why are things growing more complex? Because from microservices to CI/CD pipelines, from service meshes to blue/green deployments, all these new practices and new technologies help us make our applications and infrastructure more resilient and scalable. The downside is the ever-increasing complexity, which is actually accelerating. 

    In addition, software has officially taken over every layer of the networking and hardware stack (which sounds like an oxymoron, but it’s containers and virtualization turtles all the way down until you hit bare metal). Software is, by definition, easy to rewrite. Thus, the ease of changing software has further increased the number of changes we make to systems. Which adds still more complexity. In fact, software only ever increases in complexity; never decreases in complexity. Software is always trying to do more and encompass more. For that reason, even when you try to simplify software, you just move the complexity around rather than eliminating it. 

    Introducing Turbulence

    Here is how Netflix, the organization that pioneered chaos engineering, defines the practice: “Chaos engineering is the discipline of experimenting on a distributed system to build confidence in the system’s ability to withstand turbulent conditions.” Turbulence is the opposite of order, but it’s where all systems end up.

    In the beginning of a system’s life cycle, there is always an orderly plan for deployment and operations. You always have a clean diagram of your resources, code repositories, containers and always a nice 3D diagram of AWS instances you will deploy to and the network topology. 

    In reality, our systems are never this simple. After the system goes live, that is when we start learning about how it actually operates through unforeseen events like when marketing asks to refactor the pricing module or an outage to the payments API forces you to hardcode a token or a DNS outage—well, you know about DNS outages. In the span of a year or two, you go from 300 to 800 microservices and you have to add Kubernetes clusters and a container network interface (CNI) to properly secure the networking of those clusters. And so on and so on. The bottom line? Almost all production systems evolve into a state that we do not understand and cannot easily map back to our original systems diagram. 

    Chaos engineering is a mechanism for dealing with this growing level of complexity. The basic idea behind chaos engineering is asking the computer or application or network (or any other environmental element) a series of questions. Consider a legacy system like a mainframe. How did our legacy systems become so stable? They are well-documented. Engineers feel confident in how the system operates. The question is, was the system always that way? Is the system the same one that they originally designed? The answer is “No.” 

    Legacy systems actually went through a trial-by-fire; a constant parade of unforeseen surprises. These surprises were, effectively, questions. Through these surprises, engineers learned how the system would actually operate (or wouldn’t operate) in the face of chaotic and unforeseen events. This process created iterative feedback that helped the engineers learn how to make the system continuously more stable. 

    The point of chaos engineering is to generate this same type of feedback loop without the pain, stress and repercussions of unforeseen (and generally unwanted) events. Thus, chaos engineering is a proactive way of introducing adverse conditions to ensure that the system can still do what it’s supposed to do when unforeseen events transpire. Because everything today is software and change is constant, the questioning must also be constant. 

    For example, chances are that in the time since we originally designed any given system, it has changed a lot. This means some or all the mitigation and resilience we programmed in—retry logic, failovers, circuit breakers—are old news, designed for an older system. When triggers for those conditions actually occur, those mechanisms of resilience we designed may not perform as we had intended.  

    Chaos engineers proactively ask the question, “Under the conditions in which you are designed to perform, do you do what you are supposed to?” We introduce the conditions themselves in a controlled manner with a limited blast radius to put the system to the test and verify that our assumptions line up with the intended reality. The practice of chaos engineering is intended to be performed on live, running systems without the risk of taking down critical production assets or ambushing unsuspecting applications and their associated DevOps and development teams. This helps engineering teams reduce uncertainty by building confidence in their understanding of their systems by providing them with enhanced context. It turns out that, when engineers are presented with better context, it considerably improves their ability to make smarter, well-informed decisions where resilience is concerned. 

    Chaos Engineering for Cybersecurity

    Cybersecurity is a context-dependent discipline, dependent on how well you understand the systems you are protecting.  Software engineers are constantly changing the system. And they must have the flexibility to change it because their primary task is to constantly move towards business value. Yet to effectively apply cybersecurity practices, we must know what we are trying to secure. From that knowledge, we then can know what we most need to secure and how to do it on a given system in its most current state. 

    Before an engineer can understand how to build robust security measures that provide effective cybersecurity, the engineer must first understand the system. This reliance on the “system first” is a foundational driver for why our security systems and controls—even the immutable and ephemeral ones—experience considerable engineering drift over time. We are continuously burdened by the dependence on a stateful understanding of the system.  

    Without having some sort of testing or instrumentation feedback loop in our post-deployment world, we don’t know there is a problem with our security until there is an actual problem. Finding out that the room is on fire after the house has been burning for hours is too late. The result is the self-defeating series of reactive incident fire drills, war rooms and outages that we find ourselves in today. If we do not change this reactive crisis loop of chasing the tiger’s tail, it inevitably leaves us taking two steps back for every step we take forward with engineering teams. 

    When this drift goes undetected, we are left with our wounds exposed and attackers salivating at the meal to come.  Engineers always prefer to find evidence and sources of drift through some internal engineering failure or controlled test rather than experiencing a successful real-world attack. Over time, there is a constant drift between the system and security; engineers are constantly changing the system. As a result, there is frequently a lack of recalibration between systems and the security designed to protect them. The way to achieve this recalibration is through continuous testing or experimentation. 

    Testing is not chaos engineering. Testing is a verification and validation of something you know as either true or false. This could be vulnerability to a specific CVE or exposure to an attack pattern. In experimentation, you are trying to derive new information and explore the unknown unknowns. From this exploration, you seek to create a  better understanding of how a system reacts and then share that with the people who are designing the system.

    Live security incidents are not effective measures of detection because, by then, it is already too late in the life cycle to effectively address the problem. Worse, incidents cause stress, fear and irrational behaviors. People freak out. Not surprisingly, a security incident is not a good learning environment. Yet security incidents serve the same purpose: Asking questions about a system or systems and helping teams see how to improve them. Chaos engineering offers a better way to create these questions under conditions where it is easier to learn and observe system behaviors, providing clues to improving security. In other words, the purpose of security chaos engineering is to help us proactively understand our systems and where the security gaps are before an adversary has a chance to take advantage of them.

    There are multiple use cases for security chaos engineering including:

    • Incident response: Testing efficacy of run books, response plans, and determining whether data collection and alerting systems work as intended
    • Security control validation: Introducing conditions that should induce behavior from one or more controls and observing behavior
    • Security observability: Introducing unexpected attacks or breaches to see if your systems report and observe the incident accurately
    • Compliance monitoring: Applicable across all chaos engineering events by documenting that systems do what they say they are supposed to do

    A Practical Example

    A team I headed wrote an open source application, ChaoSlingr, designed to generate chaos engineering events for security experiments. The application consisted of three serverless functions to generate the experiment, apply it to infrastructure and then monitor the resulting impact on the system. 

    One example of how we used ChaoSlingr was an exercise where we induced an open port. Enterprises and engineers inadvertently open ports in firewalls all the time. Sometimes this is a result of misconfiguration. It may also be the result of a software upgrade after which the developer or AppSec engineer forgot to reset the port to the proper security settings. In theory, firewalls should automatically detect an unauthorized open port.

    For the experiment, we selected a set of resources that had opted-in to our experiments, selected a security group and opened the port. At the time we were relatively new to Amazon Web Services. We were surprised to find that the firewall only detected an unauthorized port opening 60% of the time. We discovered clear evidence of configuration and security tool drift. 

    We also noticed that a cloud-native configuration and management tool we had deployed was identifying and alerting the unauthorized port opening almost 100% of the time. We then realized that correlating and responding to incident events of an unauthorized port opening was complicated; it was hard to map the incident back to the log data describing how our systems responded to the event. We added a metadata pointer to each alert, allowing us to more easily correlate the information and save valuable time finding the affected security group and port. 

    In short, the exercise gave us excellent context on where our systems were drifting and failing to keep up, which of our systems were the most reliable and whether we could quickly and easily respond to these incidents by chasing down the right log files. 

    Security Chaos Engineering Programs Build Trust

    In any environment where we are managing complex systems, trust is critical. You must trust that your systems will handle unforeseen events gracefully and not cost your company money or reputational currency. You must trust that the people designing and building your systems are constantly trying to improve them.  You must trust that the security teams protecting your systems are up to the task. As systems grow more and more complex, the number of unknown unknowns multiplies. As humans, with our limited processing capacity, we cannot solve for unknown unknowns at scale, continuously. This is the task of security chaos engineering and the processes around it: To create a programmatic way to experiment against and find answers to unknown unknowns. Your systems, developers, customers and your CEO will thank you for it.

    About Secure Software Summit
    The Secure Software Summit brought together the world’s leading innovators, practitioners and academics of secure software development to share and teach the latest methods and breakthroughs on secure coding and deployment practices. If you are about developing, releasing and securing software, delivering new features fast and building things right from the start, click on the link below to get access to all of the sessions from this summit.

    https://go.shiftleft.io/secure-software-summit-2022-replay

     

  • A Closer Look at Azure DevOps

    A Closer Look at Azure DevOps

    Microsoft’s Azure DevOps Server is emerging as one of the most popular cloud application development environments. Like most continuous integration/continuous delivery (CI/CD) platforms, it provides features such as version control, reporting, project management, automated builds, and testing. It covers all aspects of the application life cycle and enables DevOps. Here’s a closer look at this cloud computing environment.

    Azure DevOps provides a number of different developer services that allow DevOps teams to plan work, collaborate on code development and build and deploy applications. It is highly collaborative and brings together developers, project managers and individual contributors to develop software. It allows organizations to create and improve products much faster than with traditional software development approaches.

    Services or Server?

    Teams can work in the cloud using Azure DevOps Services or on-premises using Azure DevOps Server.

    The cloud offering, Azure DevOps Services, is a scalable, reliable and globally available hosted service. It’s backed by a 99.9% SLA, is monitored by Microsoft’s 24/7 operations team and is available in local data centers around the world.

    The on-premises offering, Azure DevOps Server, is built on a SQL Server back end. The on-premises version is preferable when organizations need their data to stay within their network or when they want access to SQL Server reporting services that integrate with Azure DevOps Server data and tools.

    Although both offerings provide the same essential services, Azure DevOps Services offers some added benefits:

    • Simplified server management.
    • Immediate access to the new features and updates.
    • Improved connectivity with remote sites.
    • A transition from capital expenditures (servers, etc.) to operational expenditures (subscriptions).

    There are also some other key differences depending on your organization’s or your team’s needs. These can also be helpful if you’re migrating from on-premises to the cloud:

    Differences in Specific Feature Areas

    Although Azure DevOps Services is a hosted version of Azure DevOps Server, there are some feature differences. Some Server features aren’t supported in Services; for example, Azure DevOps Services doesn’t integrate with SQL Server Analysis Services to support reporting. Process customization and reporting also differ in their support.

    Integrated Features

    Azure DevOps provides integrated features that can be accessed through a web browser or an IDE client. One or more of the following standalone services can be used based on your organization’s business needs:

    • Azure Repos: Azure repos provides Git repositories or Team Foundation Version Control (TFVC) for source control.
    • Azure Pipelines: This service provides build and release services to support continuous integration and continuous delivery of applications.
    • Azure Boards: This suite of Agile tools supports planning and tracking work, code defects and issues using Kanban and Scrum methods.
    • Azure Test Plans: Azure Test Plans provides several tools to test applications, including manual/exploratory testing and continuous testing.
    • Azure Artifacts: These allow teams to share packages such as Maven, npm, NuGet and more from both public and private sources and integrate package sharing into pipelines.

    Use of the following collaboration tools is also supported:

    • Customizable team dashboards with configurable widgets to share information, progress and trends.
    • Built-in wikis for sharing information.
    • Configurable notifications.

    Azure DevOps supports adding extensions and integrating with other popular services, such as Campfire, Slack, Trello, UserVoice and more. In addition, DevOps teams can also develop their own custom extensions.

    Azure DevOps Services supports integration with GitHub and GitHub Enterprise Server code repositories. Azure DevOps Server supports integration with GitHub Enterprise Server repositories.

    Getting Started

    When you sign up for Azure DevOps, you must first register an organization, project and version control type. DevOps teams can create multiple projects and differentiate them with separate names and access permissions. Projects can be set to public or private. Teams can also use Azure Boards for collaboration and project management, and can also configure use of Agile practices. After setting up your team, you can use the platform to begin deploying your applications to Azure.

    In addition to a collaborative environment, Azure DevOps provides access to an Azure Pipeline service that supports multiple languages and platforms. The Azure DevOps pipeline is a set of services that are grouped into individual projects. 

    Additionally, Azure Test Plans feature facilitates automated testing. It is also possible to set up deployment pipelines with different components for multiple applications. 

    Azure Pipeline can track work status, manage backlogs and create custom reports. With the help of the Azure DevOps pipeline, developers can create, manage and deploy their projects in a single environment. With this solution, DevOps teams will have a single point of contact and can communicate with each other.

  • CSS-Tricks Sells Out | Peloton Spins Out | ARM Axes 1,000

    CSS-Tricks Sells Out | Peloton Spins Out | ARM Axes 1,000

    Welcome to The Long View—where we peruse the news of the week and strip it to the essentials. Let’s work out what really matters.

    This week: CSS-Tricks gets bought by DigitalOcean, Peloton pedals while its business burns, and Arm lays off up to 15% of staff. (more…)

  • Secure Software Summit: Measuring and Mitigating OSS Risks

    Secure Software Summit: Measuring and Mitigating OSS Risks

    Measuring and mitigating the security risks in open source software is becoming a major issue in the software development community. Attacks on open source software (OSS) are on the rise; open source supply chain attacks grew 650% in 2021—a staggering number.

    2021 and early 2022 saw significant attacks which impacted more than 1,000 organizations. The most prominent recent OSS attack was the Log4Shell vulnerability disclosed in December 2021. Attacks on the OSS supply chain and on OSS impacted nearly every single software organization on the planet. Everybody is scrambling to continuously patch systems as new vulnerabilities continue pouring in.

    To properly navigate this environment and keep our systems safe, we as a community need to understand the security risks of OSS and be better prepared to handle them, collectively and individually. 

    Open Source Risks

    There are a number of risks that make OSS more vulnerable without proper security measures. 

    OSS is Free; OSS Security is Expensive

    To start, we must recognize that open source software may be free to download and require no payment, but it comes with a massive security cost. 

    A significant portion of open source software is developed by volunteers in their free time. Most of the prominent open source volunteers actually have full-time jobs, and the code they contribute to projects like the Log4j logging libraries are written on nights and weekends; on their own time and not on the company dime. Not surprisingly, security isn’t high on their priority list. They don’t have the resources to invest in a secure development life cycle or have the time and bandwidth to maintain their code. This may mean that security vulnerabilities remain unaddressed for long periods of time, sometimes indefinitely. 

    All Bugs are Shallow, but There Aren’t Enough Eyeballs

    In software development, Linus’s law states “…given enough eyeballs, all bugs are shallow.” However, this is only true if those eyes are looking in the right place. In the case of open source, all code is open but there aren’t enough eyes paying attention to it to spot all the bugs. In addition, organizations may not consider the security consequences of their decision to consume a particular piece of open source software and adding it as a dependency. Dependency trees become more and more complex; harder and harder to monitor and secure, as you keep adding more third-party components to a core codebase. Each new dependency might bring additional nested dependencies—perhaps hundreds or thousands, depending on the size of the library added or the role it plays in the application architecture. For example, the Kubernetes project—one of the most successful and widely adopted OSS projects—has several thousand dependencies, many of which have additional secondary and tertiary downstream dependencies. 

    So Much Code. So Many Choices. Not all are Good.

    The beauty of the open source ecosystem is that there is a solution for everything. The problem with the open source ecosystem is that there’s not just one solution, but several dozen. For example, if you need a JSON parser for your application, there are several to choose from. If your application is running a critical service or infrastructure, then picking the right JSON parser—one that is secure and comes from a responsible project or maintainer—becomes very important. A JSON parser will serve as a critical component for your application, so it will impact the security of your entire system. If you chose the wrong JSON parser, you might pay a big price down the road in security risk and subsequent breaches. 

    Proposal: An Entry Gate Security Risk Assessment

    One potential solution to enhance OSS code security is the creation and implementation of a security risk assessment gate as part of your OSS ingestion pipeline. Putting this process in place will help you understand whether consuming a certain piece of software is high risk and avoid software that does not meet your risk criteria. 

    Here is an example of how this might work. Consider five different JSON parser libraries in GitHub, all of which have a high number of stars and appear to be highly rated. How can we assess the risk of these libraries and quickly and easily eliminate ones that are too risky? One of these JSON libraries is hosted under a user account. Most of the commits are done by a single developer without reviews. This alone indicates that using this library in a critical system presents a high risk. This lone developer could commit malicious code without review. The maintainer’s token access could be leaked, allowing attackers to insert malicious code. The developer might also turn over committer control of their library to a malicious actor without notifying users (who might not pay attention to the notification anyway). Identifying this risk required no more than a few minutes of review of the repository and cost little in time or effort. 

    A Broader View of Security Posture for OSS

    More broadly, we need to understand the full security posture of an organization or developer and their open source code prior to adding it to critical systems. Because OSS is a complex ecosystem and many pieces of OSS are complex, the risks they generate are similarly complex. Our traditional, myopic view of these risks is that we just need to find all the security vulnerabilities in our codebase and fix them. This does not resolve all risks and may lull us into a false sense of security.

    We need to understand the full comprehensive security posture of an open source project. That involves understanding a broad range of risks beyond code vulnerabilities. Here are some of the key risks to consider beyond ensuring that your codebase is well-secured and patched. 

    Supply Chain Attacks on the Build Pipeline

    Attacks like SolarWinds and CodeCove taught us that supply chain risk is a very big deal and that tools are just as much at risk as code. Attackers have found it much easier and much more reliable to attack build pipelines, rather than finding vulnerabilities in code.

    Security Risks Revealed by Continuous Security Testing 

    Continuous security testing is critical to maintaining the security of a codebase. There might not be code vulnerabilities today, but as developers continuously modify code and iterate on applications, vulnerabilities may be introduced. This can mean testing code as part of the CI/CD process with every new code push. For example, we have found numerous critical bugs in OpenSSL libraries. All were caught on a master or main branch and never made it to a production release. 

    Project Maintenance and Capabilities

    One of the really big risks with open source is the quality and frequency of project maintenance. Issues raised may not be addressed in a timely fashion. Maintainers of a project may have a long backlog of issues, indicating that they are not actively maintaining a project. It is not uncommon for critical open source projects used in millions of applications and comprising a significant dependency to the global software supply chain to not have been updated in a year or more. In addition, maintainers of OSS projects may not know how to respond to security incidents and fix security vulnerabilities. This is particularly true if the maintainers do not work at a large organization, do not have cybersecurity or secure development experience or have never suffered through the toil and trauma of addressing a major vulnerability in their code on short notice. 

    Tools for Measuring and Addressing Supply Chain Risk

    A number of efforts are underway to create free tools to help developers and application security teams better measure and address supply chain risk programmatically and analytically. 

    SLSA—Supply Chain Level for Software Artifacts

    As a tool to help organizations better address software supply chain risks, Google created a novel framework that we call Supply Chain Level for Software Artifacts (SLSA). The project has been underway at Google for more than a decade and has been open sourced as part of the OSSF. Contributors include VMware, Citibank and Datadog, among others. Google hopes to standardize SLSA and make it an industry benchmark for evaluating supply chain risk. Right now, SLSA is focused primarily on build process risks but the hope is to expand it in the near future to other areas, such as vulnerability and maintainer risks.

    OSS Security Scorecards Project

    In order to scale the process of assessing, identifying and mitigating risks in OSS, we need robust automation. In 2020, as part of the OSSF, Google released a new project called Open Source Security Scorecard. The project aimed to automate the analysis and trust aspect when evaluating the security posture of open source code. Further, the Scorecards can provide specific security recommendations. The Scorecard process can incorporate many other automated tools, including checking against vulnerabilities using the Open Source Vulnerability (OSV) database and running dependency checks using open source dependency trackers like Dependabot. The scorecards project also makes it easier to report new vulnerabilities and get those to the right people in a project. 

    Conclusion: What is not Measured is not Mitigated

    No one denies that OSS has become crucial to the vast majority of applications running the world’s technology foundation. The span of OSS continues to grow, as more and more languages and frameworks emerge and specialization increases. No one wants to reinvent the wheel and rewrite fresh code to perform the same tasks with every new application. The model of reuse, dependency and distributed software development works and speeds innovation. That said, the open source community and the supply chain used to build OSS need a more rigorous approach to reducing risk. Given the scope of the problem, automation and standardization are helping to scale toward the stated goal. We are in the early stages of creating the tools and infrastructure to automate the risk identification, measurement, assessment and mitigation processes. Security frameworks like SLSA and new processes like entry gating new OSS code with security scorecards will direct organizations down that path and move us toward a more secure open source future.

    About Secure Software Summit
    The Secure Software Summit brought together the world’s leading innovators, practitioners and academics of secure software development to share and teach the latest methods and breakthroughs on secure coding and deployment practices. If you are about developing, releasing and securing software, delivering new features fast and building things right from the start, click on the link below to get access to all of the sessions from this summit.

    https://go.shiftleft.io/secure-software-summit-2022-replay

     

  • CircleCI Report: Pipeline Optimization Best Practices

    CircleCI Report: Pipeline Optimization Best Practices

    CircleCI today published a report that found most successful software engineering teams have average workflow durations of five to 10 minutes, with any failed run being fixed or reverted in under an hour.

    The report was based on two years of data gathered from more than a quarter-billion workflows and collected from nearly 50,000 organizations using its continuous integration/continuous delivery (CI/CD) platform. The CircleCI 2022 State of Software Delivery Report also found those teams have success rates that are above 90% for the default branch of their application and deploy at least more than once per day.

    Mike Stahnke, vice president of platform for CircleCI, said the report also makes it clear there is no correlation between team size and the level of DevOps success achieved.

    Organizations that focus on optimizing pipelines are also better able to integrate testing within their DevOps workflows, he noted.

    In general, Stahnke said more DevOps teams are relying on cloud-based CI/CD platforms versus opting to deploy and maintain infrastructure themselves. That shift also lowers the barrier to entry for organizations looking to embrace DevOps best practices to accelerate application development, he added.

    Less clear is the degree to which organizations have embraced both CD and CI. While many developers routinely employ CI to accelerate application development, CD has proven to be a more elusive goal. Each platform a DevOps team supports tends to have unique attributes that require them to rely on manual processes to deploy applications. However, as DevOps processes continue to mature, Stahnke said the CI and CD integration will become more seamless. In fact, as the number of applications that organizations build simultaneously increases, the pressure to automate CD will also rise.

    However, it remains to be seen to what percentage of organizations will opt for a tightly integrated CI/CD platform versus separate CI platforms for developers and CD platforms for IT operations teams that are more loosely coupled.

    In the meantime, organizations of all sizes are clearly more focused than ever on developer productivity. In the wake of the downturn brought on by the COVID-19 pandemic, there was a rush to launch a range of digital business transformation initiatives. The issue that many organizations are now encountering is they need to find ways to enable a finite number of developers to simultaneously build and deploy multiple applications with multiple interdependencies. Many organizations are trying to address that challenge by sharing best practices across DevOps teams that have varying levels of maturity. Few organizations have a set of standardized DevOps practices that are consistently employed across every application development team within an enterprise.

    It will be up to each organization to decide whether to compare their level of DevOps expertise to other organizations. Each application development team faces a unique set of challenges. The most important goal is simply to continuously improve at a time when demand for custom applications has never been greater.

  • What Does AIOps Mean for SREs?

    What Does AIOps Mean for SREs?

    It seems SREs are of two minds when it comes to AIOps. On one hand, AIOps’ potential is pretty exciting. By automating complex workflows and troubleshooting processes, AIOps could make your life as an SRE much easier.

    But on the other hand, some SREs may choose to view AIOps with disdain and distrust. They might think of AIOps as just another trendy buzzword that doesn’t live up to the hype and that can become a distraction from the SRE tools that really matter.

    Which perspective is right? Should SREs embrace AIOps with open arms, or should they resist marketers’ efforts to position AIOps as the latest, greatest tooling innovation in the IT industry?

    Those are subjective questions that we can’t answer definitively, but let’s at least gain some perspective by examining what AIOps means for SREs.

    What is AIOps?

    If you keep up with IT buzzwords, you’ve probably at least heard the term AIOps—which is short for artificial intelligence for IT operations—and that it refers to the use of AI and machine learning (ML) to help automate IT operations workflows.

    The big idea behind AIOps is that, by using AI and ML to perform advanced analysis of large volumes of data from IT systems, IT and SRE teams can solve complex problems more efficiently than they could using a traditional, manual approach.

    AIOps can, for example, help to surface the root cause of a performance issue in a complex, multilayered environment like Kubernetes. Or it could make recommendations about how best to resolve an incident.

    AIOps entered the IT lexicon in 2016, when Gartner coined the term. Now, six years later, it’s a relatively well-established tool domain.

    How SREs View AIOps

    Despite the fact that AIOps has been around for some time, it doesn’t yet appear that many SREs have bought into the AIOps revolution. Catchpoint found in a 2021 survey that just 7.5% of SREs reported that AIOps tools delivered high value to their organizations.

    It’s unclear exactly why there are such low rates of excitement about AIOps, but we’d speculate that there are a few key factors at play:

    • AIOps is a new term for an old idea: Many monitoring and observability tools have included at least basic AI and ML analytics features for a long time, even before tool vendors slapped the AIOps label on their products. SREs probably realize this and, to some extent, view AIOps as an effort by marketers to rebrand functionality that is not actually new.
    • AIOps is hard to implement: Setting up an AIOps tool requires integrating it with diverse data sources and customizing it to fit your workflows and environment. It’s possible some SREs view this setup work as requiring more effort than it’s worth.
    • AIOps can’t replace human insight: While AIOps tools may be useful to a point, it would be unwise to place blind trust in AIOps-based analyses or recommendations. For this reason, some SREs may believe that AIOps encourages organizations to rely too heavily on automated tools at the cost of the expert analysis and perspective that only SREs can provide. (This is similar to the fact that it sometimes makes sense to prioritize human intuition and expertise over playbooks.)

    From an SRE’s perspective, then, AIOps may appear over-hyped, overly complicated and underperforming compared to traditional approaches to SRE.

    What SREs can Gain From AIOps

    SREs’ wariness toward AIOps is valid—but only to a point. It’s important not to let suspicions about the limitations of AIOps turn into excuses not to use AIOps at all. AIOps has some value to offer to SREs, even if it’s not perfect.

    For example, AIOps can play a role in reducing toil. To the extent that AIOps tools can recognize complex patterns or discover interrelated data sets more quickly than human engineers, AIOps reduces the time SREs have to spend manually troubleshooting problems or poring over complicated information.

    AIOps also helps to enable a more proactive approach to monitoring and incident management. If AIOps tools can alert SREs to emerging issues before they would otherwise recognize them, AIOps can help the SREs to get in front of the problems before they turn into true incidents. That’s better for SREs and end users alike.

    There is also an argument to be made that AIOps can help SREs do more with fewer engineering resources. If you can use AI to automate some aspects of monitoring and incident response, you can maintain the same levels of availability and performance with fewer human engineers on hand.

    Conclusion: AI Won’t Replace SREs—But it can Help

    None of the above is to say that AIOps can replace SREs, or that it magically solves every problem SREs face. Anyone who believes AIOps is a silver bullet has bought into the marketing hype to an unhealthy degree.

    Nonetheless, AIOps tools do offer value to SREs by making their jobs easier in some respects and improving reliability outcomes.

    So, while it’s wise to maintain a healthy perspective about the limitations of AIOps, SREs shouldn’t rule out AIOps tools as one way to improve reliability engineering.