devopsy.kinsta.cloud

Category: DevOps Practice

Securing APIs at the Speed of DevOps

In the 2021 State of DevOps Report, 83% of IT decision-makers told Puppet that their organizations were in the process of implementing DevOps practices to improve the quality of their software, the speed of their delivery and the security of their systems. Those DevOps organizations varied in their stages of evolution, however. For example, respondents in the middle stages of their journey reported that culture blockers such as unclear responsibilities and insufficient feedback loops were making it difficult for them to continue their DevOps journeys and join the ranks of high-evolution organizations. Such obstacles help to explain why 97% of high-evolution organizations had already applied automation to their competitive tasks compared to just two-thirds of mid-level and a quarter of low-evolution teams.

The Importance of APIs to DevOps

The prevalence of automation ties into the importance of application programming interfaces (APIs) for any organization’s DevOps journey. In the words of IBM, an API “enables companies to open up their applications’ data and functionality to external third-party developers, business partners and internal departments with their companies.” APIs, therefore, enable the services and products of different companies to communicate with one another in a way that streamlines and/or augments functionality for the user.

That said, users aren’t the only ones who benefit from APIs. DevOps teams do, too. Many of these team members deal with increasingly automated processes as part of their daily tasks, so they need APIs that can help them to deploy and configure their infrastructure. APIs can also help DevOps teams gain detailed information about an app during runtime. DevOps staff can use these insights to proactively address issues before they lead to disruption.

Security Obstacles of APIs and DevOps

Despite their benefits, APIs and DevOps aren’t without their security challenges. Contemporary Computer Services, Inc. (CCSI) identified several security hurdles that organizations may encounter along their DevOps journeys. These include the following:

A fast development process: DevOps introduces a faster speed of software delivery. In their efforts to keep up with this demand, developers might make coding mistakes that leave applications vulnerable to attack. Absent a timely security review, those vulnerable products and services might make it into production, granting attackers an opportunity to exploit the flaws and to try to access affected customers’ digital assets.

Poor collaboration: No DevOps journey will succeed without collaboration between development and operations. When there’s a lack of joint processes, these teams may stick to their silos. This way of working may leave security gaps if development and operations aren’t communicating about the management of their credentials, tokens, SSH keys and other secrets, for example.

Those security challenges can then spill over into APIs. The speed of development along with poor team communications may give rise to multiple versions of the same API. This redundancy contributes to a sense of what’s known as API sprawl, a phenomenon where the number and reach of organizations’ APIs grow exponentially. API sprawl comes with its own obstacles including maintenance difficulties and broken clients, issues that provide attackers with more opportunities for infiltrating the organization’s systems. This reality helps to explain why 91% of enterprises experienced an API security incident in 2020 and why malicious API traffic increased by 300% in the middle of 2021.

How Can Organizations Address These Security Challenges?

For an answer to this question, we look back to what highly evolved DevOps organizations are doing. Most organizations are focusing on DevSecOps. In Puppet’s report, 51% of respondents said that they are integrating security into their DevOps requirements, with even more saying the same about design (61%) and build (53%). An additional 52% of survey participants indicated that they’re bringing security into their testing. By contrast, mid- and low-evolution organizations mainly engaged in security when they were in the process of preparing for an audit of their production systems or when they learned of an issue affecting production.

Beyond those areas of focus, organizations need to emphasize runtime protection. Salt Security agrees with this statement, noting how organizations need to “security test your APIs, but know that you will also need runtime protection to catch changes that don’t go through standard build process and abuses that testing tools aren’t designed to find.”

Finally, organizations need to treat DevSecOps as the cultural change that it is. They can do this by assigning security responsibility to one person from their DevOps team. Doing so will help to ensure that teams won’t overlook security during their jobs. Simultaneously, organizations can conduct security awareness training for their developers and regularly invite both development and operations to share their stories, priorities and successes with one another. This will help to foster collaboration, thereby minimizing instances of API redundancies and API sprawl across their systems.

March 23, 2022
Turnover, Documentation and Missing Links

When employee turnover spikes, IT in general and DevOps teams in particular need to be prepared.

At the time of this writing, the stock market is roiling, and DevOps compensation is bouncing up. This has an impact on turnover because many in DevOps (and nearly all senior positions) have compensation that includes stock—stock that may be down significantly or that is currently at high risk of going down. This causes two types of movement—those leaving an organization to avoid further loss in their compensation and those who are leaving their current gig to join an organization whose stock is down. Combined with those who leave for a role with a larger base salary, there are those who are leaving to earn more and those who are leaving to avoid more loss. IT budgets ballooned during the pandemic but those, too, are deflating, so while we have a worker, not a position shortage, there will be pressure from both sides.

So, why is this geek going on about employment, compensation and available positions?

Simple: Because DevOps is not any better at documentation than any other flavor of IT. In tightly-run shops, documentation is available and maintained. In most shops, it is spotty at best. In some, it is downright nonexistent. This has forever been true in IT and, if anything, it is more true because the early stages of DevOps implementation result in a massive amount of change.

People leaving in an environment with less-than-stellar documentation can be a problem. What tools are used? Where are they located? What’s the login for these SaaS tools? Etc., etc., etc. If turnover means you’re losing entire DevOps teams in relatively short order, this can become a nightmare relatively quickly. And that nightmare is very real for everyone left behind and those that are coming in. Someone has to go figure it out; IT management has to let the rest of the business know that retention issues are causing backlogs. Everyone not tasked with figuring it out has to pick up slack for those who are … you get the picture.

Take the hour or four and document what you have. It doesn’t have to be comprehensive, it just has to list what tools, where they are, who at the organization is responsible for credentials at any SaaS vendors and under what circumstances the tools are used. If each team documents their own stuff, even in a shared toolchain environment, comparing the documents will uncover little details one missed that another did not.

Let’s face it, we have—for better or worse—moved largely to a world where loyalty is based on compensation. Much like mercenaries, most IT employees are always listening for a better offer. So the organization, while acknowledging this fact, needs to take steps to minimize the negative impacts of turnover. New ideas and suggestions to change things are all good and you don’t want to interfere with that aspect of turnover. But anything that impedes the organization’s ability to deliver should be addressed beforehand. And that means documentation so remaining and new employees can get up to speed quickly and take ownership rapidly and with confidence.

And keep rocking it. Whether you are looking for a new gig or not, keep the apps you are responsible for running—and document them. You’ve contributed to the success of your organization; don’t slack off while moving along. And don’t hate those that are moving along. Not all of us base our loyalty on compensation, but we do all allow compensation to influence our decisions (unless you are working for free and I missed it?) Keep rocking, no matter your status. And don’t hesitate to tell interviewers that you helped your last organization achieve success.

March 23, 2022
Apple Outage Outrage | Linux Random Redo | Okta Hacked (or Not)

Welcome to The Long View—where we peruse the news of the week and strip it to the essentials. Let’s work out what really matters.

This week: Why Apple services were down, Linux gets a huge RNG overhaul, and we wonder if Okta was hacked again. (more…)

March 22, 2022
Secure Software Summit: Behold the SBOM

With supply chain security becoming more of a focus, the SBOM is now viewed as a critical element in shoring up supply chain security. SBOM stands for software bill of materials. At a very elementary level, an SBOM is a list of ingredients. Think about how every food product in the supermarket lists ingredients so, for example, people who are allergic to nuts can avoid nuts. An SBOM does the same thing for software applications.

If you want to do business with the U.S. government, SBOMs are no longer optional. Executive order 14028 requires an SBOM for all U.S. federal software procurements. Why the push for SBOMs now? Software companies generally do not list all the components included in the software they produce. Knowing what components are included in applications is essential for proactive vulnerability management. That said, the government mandate for SBOMs only requires a minimal amount of included information—supplier name, the component name and version, dependency relationships, time stamps for versions or modification and information about who authored the SBOM itself.

What to Look For in an SBOM

This minimum required data yields minimal benefit for cybersecurity risk reduction and vulnerability management. SBOMs can actually include many other types of information. What information should you include in SBOMs for software applications produced, purchased or consumed by your organization? Here is a list.

Authenticity —Authenticity asks and verifies that someone actually created a listed component. Ideally, this requires a digital signature of some sort. Authenticity can be particularly important in the case of nested components where the supplier of an application includes multiple other components, as is common with JavaScript and Java libraries used for common functionalities.

Integrity —This addresses whether a component has been tampered with or modified without authorization. An SBOM should include some measure of integrity. Equally important is proving the integrity of the SBOM itself. Just as CheckSum can tell you whether software you are loading matches what the source of that software indicates, integrity checks should allow you to verify that components on an SBOM (and the SBOM itself) are exactly what you are expecting.

Services —An inventory of services called by an application is critical. Modern software does not live in a vacuum. It calls out to the Internet to perform all sorts of tasks and communication with other systems. Those services that extend beyond your application are part of the inventory because they can be an indication of vulnerability or indicate a larger problem.

Vulnerability Remediation Status —Development teams are constantly balancing the need to remediate vulnerabilities against the risk that remediation will break the application. Some open source projects, for example, have a history of not backporting security fixes. The latest version of the project may break applications. Organizations are forced to choose between breaking an application or taking the risk of introducing a high-severity vulnerability that might be exploited. Responsible organizations that opt to keep the older version and not break their software will backport their own fixes. An SBOM should describe these variations and any other vulnerability remediation activity.

Inventory of component security advisories —An SBOM should describe security advisories pending against all components. This is likely to be a dynamic list that you will need to automatically update using one of the accepted vulnerability advisory formats (CSAF and CycloneDX, to name two). In fact, this is one of the strengths of SBOMs—they are designed for automation and to act programmatically.

Completeness —It is not always possible to ship complete SBOMs, either for technical or intellectual property reasons. An SBOM should indicate whether it is complete and whether it contains a comprehensive inventory of dependencies and services.

Pedigree —This information describes the DNA of the SBOM and the original makeup and changelog of each particular component. An SBOM should describe the original component, exactly what changes were made, who committed the changes and why, what the differences were between different versions and what security issues were fixed with each variation and version.

Provenance —This relates to the origin of a component or service. It can describe a repository that the component was downloaded from, a country of origin, the name of a software manufacturer or open source project or other information indicating the component or service’s source of origin.

License type —License information is relevant for compliance and to ensure that your organization is not consuming or producing software in a manner that violates licensing terms. Moreso than for security and vulnerability, this information is for risk management, compliance enforcement and auditing.

Customizing SBOMs to Meet Your Needs

The SBOM is a flexible and highly adaptable approach that can be easily customized to fit specific use cases. You can design SBOMs to meet the specific needs of your use case, either as a software provider or consumer. You also may need to modify your SBOM fields and data design as your needs evolve. Fortunately, it is rapidly becoming easier to automatically generate SBOMs using software security tools such as software composition analysis (SCA) and interactive application security testing (IAST) solutions. SBOMs can also be created or read automatically at other points during the build life cycle along the continuous integration/continuous deployment (CI/CD) process. Just as you would not buy a box of cereal that did not list the ingredients, in the future you will not want to consume software that does not have a detailed SBOM. What you don’t know about the software you run can hurt you, and SBOMs will be a crucial tool in providing greater transparency and observability into the software supply chain.

About Secure Software Summit
The Secure Software Summit brought together the world’s leading innovators, practitioners and academics of secure software development to share and teach the latest methods and breakthroughs on secure coding and deployment practices. If you are about developing, releasing and securing software, delivering new features fast and building things right from the start, click on the link below to get access to all of the sessions from this summit.

https://go.shiftleft.io/secure-software-summit-2022-replay

March 22, 2022
Your DevOps Teams Can Do Better. Here’s How.
As more organizations focus on digital transformation as the centerpiece of their growth strategy, software developers and IT teams are under heightened pressure to perform—quickly and efficiently.

That begs two questions: Can we do better? And if so, how can we do better?

The answers can be elusive, especially for DevOps teams that have over-relied on processes and models that, while previously successful, may not be as relevant for today’s projects. Of course, the point about relevance has been further magnified during the pandemic, when sudden shifts to remote working left many teams scrambling to ensure they had the capacity and infrastructure to support massive work-from-home cohorts.

Software development and IT operations support teams that are winning by doing better in this new era are truly embracing Agile. And that means taking to heart the definition of “Agile,” which is a mindset of values and principles to continually evolve your way of working (WoW) to ensure your productivity and value are tailored to fit the tasks at hand.

Conversely, teams that have relied on their go-to framework or approach for every assignment are often missing the boat with respect to efficiencies, effectiveness and even ROI. This is because they aren’t assessing whether their proven WoW is useful for their current projects.

That can lead to underwhelming results. A report from Digital.ai revealed that while over 90% of software development teams take an Agile approach to their work, a whopping 96% of their digital transformation projects have failed or been delayed. Clearly, that suggests Agile has worked for them in the past, but the nature of digital transformation projects is calling for teams to adapt to new or hybrid approaches that may differ from their go-to WoW.

Challenging? Yes. Doable? Also Yes

While there is no quick fix to attain true flexibility amid complexity, the journey begins with the recognition that, yes, you can do better. And to do better—to become an adaptive, lean, responsive and learning organization under any circumstances—there are a number of steps to take to ensure your journey to Agile freedom is successful.

Here are five that I believe must be considered for DevOps teams to become better and more effective:

Consider all options. Agile has been practiced for more than two decades, and literally hundreds of frameworks and approaches have been developed and implemented. If you examine your WoW in a context-driven, pragmatic manner you are more likely to select the best-fitting technique given the situation.

Do your homework. Your team leaders may not be familiar with the breadth of the Agile, traditional and hybrid frameworks at their disposal. But there’s help available. Seek out online courses and certifications designed to help organizations and individuals explore and navigate the multitude of Agile frameworks that they may currently be overlooking.

Focus on value. It’s great if your gut tells you that your Agile framework is reaping real results, but can you prove it? If your senior leadership has invested in Agile training, coaching and certifications, they’ll demand to see some tangible returns. With well-chosen Agile approaches that match the work required, you will be able to deliver the proof that Agile is driving measurable value that matters. Even further, learning which metrics to use in different situations can help measure this value.

Sidestep bottlenecks. By choosing the best WoW for the project at hand, you are more likely to eliminate the wasteful or inefficient processes that can add hours or days to any assignment. Understanding how to unclog these pipes is critical in improving the flow of value to your customers.

Motivate with morale. If your team is frustrated and morale is low, productivity, the quality of the work and your retention rate will be, too. But, if you’re able to do Agile well, this can help your teams be happier, more productive and more successful in accomplishing project goals. Ultimately, that means a more joyful customer base, too.
Conclusion

In the long run, the framework or approach you choose isn’t as important as your willingness to be open-minded in applying Agile frameworks to projects on a case-by-case basis. As you become more comfortable with new or different approaches, you will promote continuous improvement through carefully considered experimentation.

That means you’ll become a learning organization, one that is open to change and evolution. With that mindset, your development and IT ops team will do better—guaranteed. Because true agility comes from freedom, not frameworks.
March 22, 2022
Survey Sees App Testing More Widely Distributed

A survey of 900 application testing professionals conducted by PractiTest, a provider of a platform for managing testing tools, suggested the number of organizations that have dedicated testing teams is declining at a steady rate.

The survey found a 15% decrease in organizations with dedicated testing teams that have six or more members. At the same time, however, the survey found only 18% of respondents reported that all testing is done by individuals that are not part of a dedicated testing team.

Joel Montvelisky, test specialist and chief solution architect at PractiTest, said that shift indicates that more organizations are steadily incorporating testing within DevOps workflows. In fact, more than 60% of respondents said DevOps and Agile processes led to teams being more organized and stable, with two-thirds (66%) reporting that development teams were releasing features more frequently with more functionality.

However, only 38% of respondents said they were active participants in a continuous integration/continuous delivery (CI/CD) workflow. A full 90% said they needed to coach or train other members within the team and just over half (54%) reported that estimations of the time required for development and testing were accurate.

Two-thirds also said collaboration with developers is also improved when applying DevOps processes, with more than half reporting fewer serious bugs made it into production. More than half also said they monitor software in production and deployment environments. The number of testers who defined themselves as engineers has gone up by 14% year-over-year. However, only 44% report that automation has replaced 50% or more of their manual testing efforts.

A full 95% or more of survey respondents identified application programming interface (API) testing, Agile methodologies, functional testing and scripting as high on their list of priorities. In contrast, big data testing, internet of things (IoT) testing and machine learning/artificial intelligence (AI) testing were cited as important or very important by only 60% of respondents.
Finally, about a third of survey respondents noted that non-testers are now involved in testing activities.

Montvelisky said the PractiTest survey makes it clear that while developers are a little more involved in testing than they were a few years ago, organizations still relied on testing specialists that are becoming more integrated into DevOps workflows. This is especially important as more organizations launch digital business transformation initiatives that require them to build software using the same types of DevOps processes that independent software vendors (ISVs) employ.

It’s still too early to say just how far testing might shift left toward developers, but as testing becomes more automated—thanks, in part, to increased reliance on AI—the types of tests that will be conducted by a dedicated testing team will evolve. In fact, it’s unlikely automation will eliminate the need for those teams any time soon. Rather, testers will be augmented by AI technologies that will help them to keep pace with the accelerated rate of application development enabled by wider adoption of DevOps best practices within organizations both large and small.

March 21, 2022
Secure Software Summit: Reachability and Risk for Security Leaders
It is impossible to manage security posture without considering two key factors in any potential vulnerability or security flaw: Reachability and risk. The two factors are related. Reachability defines the degree to which a given security vulnerability that is detected, such as a CVE, can actually be attacked and exploited to gain privileged access and directly or indirectly access critical systems or data. Risk is a business measurement that assesses the potential for a vulnerability to actually damage an enterprise or organization. In general, without reachability, there is less risk.

How to Think About Reachability

Reachability and risk have dramatically shifted in recent years as we have moved from a hard perimeter “fortress” approach to security to more distributed systems, APIs, cloud computing, SaaS and far more potential points for attack ingress and exfiltration egress. To mitigate this new risk landscape, we have actually moved to fortify all the newly exposed internal elements and the critical systems of security. We have built a new perimeter around identity, with authentication and zero-trust. We have built perimeters around connected devices, around our cloud infrastructure and our virtualized environments and our configuration of all these systems. At every point where there is a potential for risk, you need to think about fortifying the new mini-perimeters.

A key aspect of risk and reachability that we have spent less effort on is the human aspect. We do not yet consider people as a perimeter to be defended, even though people comprise the most reachable vulnerability. As people, we create flawed software. We misconfigure our infrastructure, our security tools and our authentication systems. A key part of the new mindset of reachability must be better accounting for and mitigating the risks of people. This means simplifying their decision trees, either through better systems or better design, and by helping them become a better gray-matter perimeter.

Due to the atomization of our IT and application environment, and the resulting miniaturization of our security fortresses, we have created a far more granular surface. In some cases, such as through segmentation, we have enhanced security and reduced risk; in other cases, we have added complexity. Without a doubt, we now have many more points of contact and a broader attack surface exposed to the world. A critical consideration when assessing reachability is how far each of those points can be exploited, and to what breadth and depth—looking at horizontal traversal and immediate compromise of critical systems or subverting supply chain and other pivotal subsystems for subsequent attacks. The miniature fortress mentality is a useful framework for thinking through the potential attack surface and putting in place the right controls, at the right places, to address reachability.

Risk Scoring and Reachability

So what is the interplay between risk scoring and reachability? We cannot say no to new technologies that introduce more complexity, and potentially more risk, if our end users are demanding them. We need to better manage how risk scores are created and applied to help people perform better in this environment of greater complexity.

Today, the responsibility for understanding the complete security picture and applying the proper context to decision-making and judgments has grown more widespread. Everyone who is deploying, using, building or designing applications and technology environments must be constantly thinking about the security implications of whatever project or code they are working on. The lens for those implications is not only at the application or system or infrastructure level but for the general well-being of the business. This means a broader understanding of risk scoring driven by reachability and criticality and taking into better account the implications of reachability and attackability.

In cybersecurity, there is no shortage of risk measurements. Injecting reachability into the scoring process is a way of framing vulnerabilities to prioritize which of these issues are likely to be an actual risk. It is possible to have something that comprises a major business risk but have a very low reachability context, which may lead a security team to treat that vulnerability differently.

Broadly speaking, however, it is insufficient to make decisions solely based on programmatically generated risk scores. Any risk score is a reflection of the system generating the score, and how that system was tuned, either by the vendor or by your team. It is important to always remember that any risk score involves some subjective judgments. This is necessary, but injects opportunities for drift and bias that may skew scores in ways that leave you less secure.

To be clear, risk scores serve a crucial purpose in application security. Risk scores help you reduce the noise and find signals amidst the flood of CVEs and vulnerabilities and alerts. In fact, there is today so much noise around security that any effective team relies on multiple layers of noise reduction in their decision chain and prioritization and policy efforts. Where risk scores fail is with the inability to map accurately to business risk or by inaccurately reflecting the reachability of any CVE or other reported issue. This requires security teams to build their own internal and mental models of risk, modeling threats against the most critical systems and creating context that can be used to verify decisions by the risk scoring systems. Knowing how data flows through applications and infrastructure is crucial if a human agent can quickly judge the reachability and attackability of any given vulnerability presented or recognize when the automated risk scoring engines may not accurately reflect business risks.

In effect, there are two different processes at work: Calculating risk and contemplating risk. Calculating risk is programmatic and automated; human-guided, initially, but driven by models applied against the mass of vulnerabilities surfaced by security testing tools like software composition analysis, static application security testing, fuzzing and linting. Contemplating risk is a more holistic exercise that leverages the human capacity for intuition and innate knowledge of systems driven by experience and hard-won insights.

Contemplation is what will help security teams identify undetected biases, guide threat modeling to better consider drift and highlight areas where human breakdowns might have augmented risk in ways that a risk scoring system could never capture. Above all, contemplation is about context. The more context a human actor has, the better they can understand the big picture. The greater the understanding of the big picture, the better decisions people make. This must also flow through and inform the models and the risk scoring engines so that the value of contemplation is captured in the calculation.

Using Risk Scoring Tools to Minimize Reachability and Attackability

A security team must constantly measure the efficiency and efficacy of all security tooling. The general rule is that security controls lose efficacy over time as vendors lose focus or fail to account properly for emerging classes of threats. When you have a security tool that generates risk scores in tune with your reachability guidelines and proves accurate, you should elevate that system to a higher level of trust and give more weight to its findings. That said, even tools that generate risks you think are accurate and perform well must be constantly evaluated and cross-checked.

Unfortunately, it only takes a small amount of drift to open up an organization to potentially catastrophic attacks. A critical part of this calculation is ensuring that humans stay on top of how the risk scores are created and continue to influence the scores to fight drift. Drift is natural but it is not inevitable. Your code and environment are constantly changing, This may result in changes in reachability which can materially impact your overall risk posture and should alter the way you approach different systems and risks. An automated risk scoring system may not understand that a new type of supply chain attack might result in subsequent related attacks on similar components in other operating systems, to name one example. Human intelligence and contemplation, really, is the secret ingredient that makes risk scoring tools far more valuable and reliable.

Takeaways: Atomized Systems, Reachability and Risk and Humans

The environment and application landscape will continue to atomize further as we move into new infrastructure paradigms like edge computing and all systems on the network gain capacity. For this landscape, security teams and the developers they work with on shifting left should:
- Recognize that we have shifted from a global perimeter to an atomized constellation of mini-perimeters and shift their mindset to consider security and building fortresses at the micro level as a key way to reduce reachability and risk
- Developers and security teams need to assess the systems they build and protect on the basis of reachability and risk to properly prioritize. Risk scoring tools are useful elements for reducing noise but cannot supplant human intelligence to gut-check risk decisions
- All along the way, humans have to continue to guide and inform these risk scoring systems and ensure that risk scores do not become blind proxies for criticality and importance. Allowing automated risk scoring to dominate the decision process can make security drift more dangerous and allow human error to go unrecognized, leaving applications and infrastructure both reachable and at risk.
This article was co-authored by Bryan Smith, CTO, RiskLens and Rob Lundy, director of product marketing, ShiftLeft.

About Secure Software Summit
The Secure Software Summit brought together the world’s leading innovators, practitioners and academics of secure software development to share and teach the latest methods and breakthroughs on secure coding and deployment practices. If you are about developing, releasing and securing software, delivering new features fast and building things right from the start, click on the link below to get access to all of the sessions from this summit.

https://go.shiftleft.io/secure-software-summit-2022-replay
March 21, 2022
5 Steps to More Sustainable DevOps
The rise of remote workforces, cryptocurrencies and super-sized data centers brings to the forefront a concern in tech development that doesn’t get enough attention: The impact of computing and advanced technologies on the environment.

About six months into the pandemic, people began to notice cleaner air, clearer water and an overall calming effect on the environment in the reduced presence of industry. Memes were created and widely circulated that suggested maybe we were the problem after all.

That lasted for about six weeks before cabin fever set in, and we became impatient and ready to get back to business as usual.

In the time between the last quarter of 2020 and the sudden waxing and waning of NFTs, a few timid voices again raised concerns about the impact of super-computing and other technologies on the environment. Those concerns eventually became muffled amidst socioeconomic and political turmoil, fears about inflation, staff shortages and logistical issues surrounding supply chain disruption.

App development tried to fill some of those gaps and new data centers appeared as we struggled to find our footing again in a post-pandemic world.

While environmental issues have been a matter of growing concern for decades, the past few years have put a particularly intense focus on the issue, with entire industries growing up around everything from climate-friendly investing to carbon offsets, and more. The environmental impact of technology has become an issue yet unresolved in spite of a growing focus on sustainability.

Addressing the Elephant in Your Staging Area

The typical DevOps journey focuses mainly on how app development will impact an organization. Rarely does it address how development impacts the environment.

In addition to factors like improving time-to-market, reducing human error, debugging and making financial resource management more efficient, part of your DevOps strategy should include ways to reduce your carbon footprint and impact on the planet.

First, it helps to understand exactly how technology affects the climate and depletes natural resources so you can incorporate sustainable development into your plan in a meaningful way.
- Carbon emissions: It’s estimated that in 2022, there are already more than 16.5 billion mobile devices in use worldwide. Just one iPhone emits up to 86kg of CO2 over its life cycle, and 80% of that occurs before it even gets into the hands of consumers. This is just the tip of the carbon emissions iceberg, and it will only get worse.
- Power consumption: All of that carbon comes from somewhere, and taking a look at energy consumption from just one slice of technology provides a glimpse into just how heavily computing contributes to the problem. As of 2020, data centers used about 416 terawatts of energy annually. The Bitcoin network alone consumes more power than all global data centers combined. A single NFT transaction consumes 82 kWh of power and emits 48 kg of CO2.
Even relatively passive use of technology, like video streaming and remote collaboration, for example, has the potential to consume more power than entire countries. Add up all of the hours of Netflix, Zoom meetings and other remote activities and ponder the implications.
- Waste: Despite the number of devices in use, very little hardware is recycled. Electronic waste comprises 70% of toxic waste worldwide, and it’s estimated to total 40 million tons annually.
- Water consumption: You don’t often consider mixing water and electricity, but all of those data centers need to be cooled somehow. Internet usage alone uses 3,000 liters of water per person each year and emits 2,000 kg of CO2.
Cleaning up Your DevOps Process for the Good of the Climate

To become part of the solution, it’s important to do more than just invest in green businesses. Here are some steps you can take to clean up your DevOps process and become part of the solution rather than contributing to the problem.

Identify Areas of Vulnerability

Software engineering and DevOps are increasingly embracing new territory and emphasizing sustainable engineering. The first step in the right direction is to become aware of how power and resources are consumed during your development cycle.

Consider:
- How your power is generated. For example, are you dependent on fossil fuels or more eco-friendly methods?
- How often are you replacing or recycling hardware and other physical components? This is a goal in waste reduction.
- How are you cooling data centers and computing equipment? Are your efforts at maintaining optimal temperatures wasting resources?
Balance the Three Areas of Sustainable DevOps

In the realm of sustainable engineering, the process can be broken down into three categories: Technical, operational and environmental. The goal is to balance these aspects to monitor and optimize the development process and do the least amount of damage possible.

Enforce Sustainable Development Practices

Once you’ve identified areas of risk and set goals for sustainable software engineering, use the results to create and enforce a set of development best practices that better support environmentally responsible design and production. If you need some help or ideas, you can model your own initiatives after these emerging standardization principles.

Become Part of the Green Tech Community

In 2021, more than 200 countries signed the Glasgow Climate Pact, solidifying their commitment to reducing their carbon footprint and resultant environmental damage from tech development and use. The conference brought together stakeholders from businesses, government agencies and others to share information and create standards and best practices for sustainable computing and tech development.

Incorporate Tools That Support Sustainable Development

Tech giants like Google, Amazon, and Microsoft are creating tools to help DevOps and IT monitor and measure the environmental impact of everything from cloud computing to app development. Seek out and incorporate these tools into your own set of best practices or work on creating your own solutions.

Final Thoughts

Tech development is intended to improve our quality of life and reshape how we conduct business. There’s no reason that intention shouldn’t extend to the world in which we live and engage in commerce.

The first step toward combating climate change without negatively impacting development is understanding how technology affects our environment. Then, we can make informed decisions that support responsible, sustainable DevOps.
March 21, 2022
Secure Software Summit: Applying Chaos Engineering to Software Security
Today’s software systems are, essentially, controlled chaos—and lightly controlled chaos, at that. This makes it exceptionally challenging to model the behavior of those systems. Our systems are quickly becoming larger and larger, with more and more moving parts. It is not uncommon for enterprises to have over 1,000 microservices and millions of containers running thousands of applications containing many thousands of open source libraries.

Why are things growing more complex? Because from microservices to CI/CD pipelines, from service meshes to blue/green deployments, all these new practices and new technologies help us make our applications and infrastructure more resilient and scalable. The downside is the ever-increasing complexity, which is actually accelerating.

In addition, software has officially taken over every layer of the networking and hardware stack (which sounds like an oxymoron, but it’s containers and virtualization turtles all the way down until you hit bare metal). Software is, by definition, easy to rewrite. Thus, the ease of changing software has further increased the number of changes we make to systems. Which adds still more complexity. In fact, software only ever increases in complexity; never decreases in complexity. Software is always trying to do more and encompass more. For that reason, even when you try to simplify software, you just move the complexity around rather than eliminating it.

Introducing Turbulence

Here is how Netflix, the organization that pioneered chaos engineering, defines the practice: “Chaos engineering is the discipline of experimenting on a distributed system to build confidence in the system’s ability to withstand turbulent conditions.” Turbulence is the opposite of order, but it’s where all systems end up.

In the beginning of a system’s life cycle, there is always an orderly plan for deployment and operations. You always have a clean diagram of your resources, code repositories, containers and always a nice 3D diagram of AWS instances you will deploy to and the network topology.

In reality, our systems are never this simple. After the system goes live, that is when we start learning about how it actually operates through unforeseen events like when marketing asks to refactor the pricing module or an outage to the payments API forces you to hardcode a token or a DNS outage—well, you know about DNS outages. In the span of a year or two, you go from 300 to 800 microservices and you have to add Kubernetes clusters and a container network interface (CNI) to properly secure the networking of those clusters. And so on and so on. The bottom line? Almost all production systems evolve into a state that we do not understand and cannot easily map back to our original systems diagram.

Chaos engineering is a mechanism for dealing with this growing level of complexity. The basic idea behind chaos engineering is asking the computer or application or network (or any other environmental element) a series of questions. Consider a legacy system like a mainframe. How did our legacy systems become so stable? They are well-documented. Engineers feel confident in how the system operates. The question is, was the system always that way? Is the system the same one that they originally designed? The answer is “No.”

Legacy systems actually went through a trial-by-fire; a constant parade of unforeseen surprises. These surprises were, effectively, questions. Through these surprises, engineers learned how the system would actually operate (or wouldn’t operate) in the face of chaotic and unforeseen events. This process created iterative feedback that helped the engineers learn how to make the system continuously more stable.

The point of chaos engineering is to generate this same type of feedback loop without the pain, stress and repercussions of unforeseen (and generally unwanted) events. Thus, chaos engineering is a proactive way of introducing adverse conditions to ensure that the system can still do what it’s supposed to do when unforeseen events transpire. Because everything today is software and change is constant, the questioning must also be constant.

For example, chances are that in the time since we originally designed any given system, it has changed a lot. This means some or all the mitigation and resilience we programmed in—retry logic, failovers, circuit breakers—are old news, designed for an older system. When triggers for those conditions actually occur, those mechanisms of resilience we designed may not perform as we had intended.

Chaos engineers proactively ask the question, “Under the conditions in which you are designed to perform, do you do what you are supposed to?” We introduce the conditions themselves in a controlled manner with a limited blast radius to put the system to the test and verify that our assumptions line up with the intended reality. The practice of chaos engineering is intended to be performed on live, running systems without the risk of taking down critical production assets or ambushing unsuspecting applications and their associated DevOps and development teams. This helps engineering teams reduce uncertainty by building confidence in their understanding of their systems by providing them with enhanced context. It turns out that, when engineers are presented with better context, it considerably improves their ability to make smarter, well-informed decisions where resilience is concerned.

Chaos Engineering for Cybersecurity

Cybersecurity is a context-dependent discipline, dependent on how well you understand the systems you are protecting. Software engineers are constantly changing the system. And they must have the flexibility to change it because their primary task is to constantly move towards business value. Yet to effectively apply cybersecurity practices, we must know what we are trying to secure. From that knowledge, we then can know what we most need to secure and how to do it on a given system in its most current state.

Before an engineer can understand how to build robust security measures that provide effective cybersecurity, the engineer must first understand the system. This reliance on the “system first” is a foundational driver for why our security systems and controls—even the immutable and ephemeral ones—experience considerable engineering drift over time. We are continuously burdened by the dependence on a stateful understanding of the system.

Without having some sort of testing or instrumentation feedback loop in our post-deployment world, we don’t know there is a problem with our security until there is an actual problem. Finding out that the room is on fire after the house has been burning for hours is too late. The result is the self-defeating series of reactive incident fire drills, war rooms and outages that we find ourselves in today. If we do not change this reactive crisis loop of chasing the tiger’s tail, it inevitably leaves us taking two steps back for every step we take forward with engineering teams.

When this drift goes undetected, we are left with our wounds exposed and attackers salivating at the meal to come. Engineers always prefer to find evidence and sources of drift through some internal engineering failure or controlled test rather than experiencing a successful real-world attack. Over time, there is a constant drift between the system and security; engineers are constantly changing the system. As a result, there is frequently a lack of recalibration between systems and the security designed to protect them. The way to achieve this recalibration is through continuous testing or experimentation.

Testing is not chaos engineering. Testing is a verification and validation of something you know as either true or false. This could be vulnerability to a specific CVE or exposure to an attack pattern. In experimentation, you are trying to derive new information and explore the unknown unknowns. From this exploration, you seek to create a better understanding of how a system reacts and then share that with the people who are designing the system.

Live security incidents are not effective measures of detection because, by then, it is already too late in the life cycle to effectively address the problem. Worse, incidents cause stress, fear and irrational behaviors. People freak out. Not surprisingly, a security incident is not a good learning environment. Yet security incidents serve the same purpose: Asking questions about a system or systems and helping teams see how to improve them. Chaos engineering offers a better way to create these questions under conditions where it is easier to learn and observe system behaviors, providing clues to improving security. In other words, the purpose of security chaos engineering is to help us proactively understand our systems and where the security gaps are before an adversary has a chance to take advantage of them.

There are multiple use cases for security chaos engineering including:
- Incident response: Testing efficacy of run books, response plans, and determining whether data collection and alerting systems work as intended
- Security control validation: Introducing conditions that should induce behavior from one or more controls and observing behavior
- Security observability: Introducing unexpected attacks or breaches to see if your systems report and observe the incident accurately
- Compliance monitoring: Applicable across all chaos engineering events by documenting that systems do what they say they are supposed to do
A Practical Example

A team I headed wrote an open source application, ChaoSlingr, designed to generate chaos engineering events for security experiments. The application consisted of three serverless functions to generate the experiment, apply it to infrastructure and then monitor the resulting impact on the system.

One example of how we used ChaoSlingr was an exercise where we induced an open port. Enterprises and engineers inadvertently open ports in firewalls all the time. Sometimes this is a result of misconfiguration. It may also be the result of a software upgrade after which the developer or AppSec engineer forgot to reset the port to the proper security settings. In theory, firewalls should automatically detect an unauthorized open port.

For the experiment, we selected a set of resources that had opted-in to our experiments, selected a security group and opened the port. At the time we were relatively new to Amazon Web Services. We were surprised to find that the firewall only detected an unauthorized port opening 60% of the time. We discovered clear evidence of configuration and security tool drift.

We also noticed that a cloud-native configuration and management tool we had deployed was identifying and alerting the unauthorized port opening almost 100% of the time. We then realized that correlating and responding to incident events of an unauthorized port opening was complicated; it was hard to map the incident back to the log data describing how our systems responded to the event. We added a metadata pointer to each alert, allowing us to more easily correlate the information and save valuable time finding the affected security group and port.

In short, the exercise gave us excellent context on where our systems were drifting and failing to keep up, which of our systems were the most reliable and whether we could quickly and easily respond to these incidents by chasing down the right log files.

Security Chaos Engineering Programs Build Trust

In any environment where we are managing complex systems, trust is critical. You must trust that your systems will handle unforeseen events gracefully and not cost your company money or reputational currency. You must trust that the people designing and building your systems are constantly trying to improve them. You must trust that the security teams protecting your systems are up to the task. As systems grow more and more complex, the number of unknown unknowns multiplies. As humans, with our limited processing capacity, we cannot solve for unknown unknowns at scale, continuously. This is the task of security chaos engineering and the processes around it: To create a programmatic way to experiment against and find answers to unknown unknowns. Your systems, developers, customers and your CEO will thank you for it.

About Secure Software Summit
The Secure Software Summit brought together the world’s leading innovators, practitioners and academics of secure software development to share and teach the latest methods and breakthroughs on secure coding and deployment practices. If you are about developing, releasing and securing software, delivering new features fast and building things right from the start, click on the link below to get access to all of the sessions from this summit.

https://go.shiftleft.io/secure-software-summit-2022-replay
March 18, 2022
A Closer Look at Azure DevOps
Microsoft’s Azure DevOps Server is emerging as one of the most popular cloud application development environments. Like most continuous integration/continuous delivery (CI/CD) platforms, it provides features such as version control, reporting, project management, automated builds, and testing. It covers all aspects of the application life cycle and enables DevOps. Here’s a closer look at this cloud computing environment.

Azure DevOps provides a number of different developer services that allow DevOps teams to plan work, collaborate on code development and build and deploy applications. It is highly collaborative and brings together developers, project managers and individual contributors to develop software. It allows organizations to create and improve products much faster than with traditional software development approaches.

Services or Server?

Teams can work in the cloud using Azure DevOps Services or on-premises using Azure DevOps Server.

The cloud offering, Azure DevOps Services, is a scalable, reliable and globally available hosted service. It’s backed by a 99.9% SLA, is monitored by Microsoft’s 24/7 operations team and is available in local data centers around the world.

The on-premises offering, Azure DevOps Server, is built on a SQL Server back end. The on-premises version is preferable when organizations need their data to stay within their network or when they want access to SQL Server reporting services that integrate with Azure DevOps Server data and tools.

Although both offerings provide the same essential services, Azure DevOps Services offers some added benefits:
- Simplified server management.
- Immediate access to the new features and updates.
- Improved connectivity with remote sites.
- A transition from capital expenditures (servers, etc.) to operational expenditures (subscriptions).
There are also some other key differences depending on your organization’s or your team’s needs. These can also be helpful if you’re migrating from on-premises to the cloud:
Differences in Specific Feature Areas

Although Azure DevOps Services is a hosted version of Azure DevOps Server, there are some feature differences. Some Server features aren’t supported in Services; for example, Azure DevOps Services doesn’t integrate with SQL Server Analysis Services to support reporting. Process customization and reporting also differ in their support.

Integrated Features

Azure DevOps provides integrated features that can be accessed through a web browser or an IDE client. One or more of the following standalone services can be used based on your organization’s business needs:
- Azure Repos: Azure repos provides Git repositories or Team Foundation Version Control (TFVC) for source control.
- Azure Pipelines: This service provides build and release services to support continuous integration and continuous delivery of applications.
- Azure Boards: This suite of Agile tools supports planning and tracking work, code defects and issues using Kanban and Scrum methods.
- Azure Test Plans: Azure Test Plans provides several tools to test applications, including manual/exploratory testing and continuous testing.
- Azure Artifacts: These allow teams to share packages such as Maven, npm, NuGet and more from both public and private sources and integrate package sharing into pipelines.
Use of the following collaboration tools is also supported:
- Customizable team dashboards with configurable widgets to share information, progress and trends.
- Built-in wikis for sharing information.
- Configurable notifications.
Azure DevOps supports adding extensions and integrating with other popular services, such as Campfire, Slack, Trello, UserVoice and more. In addition, DevOps teams can also develop their own custom extensions.

Azure DevOps Services supports integration with GitHub and GitHub Enterprise Server code repositories. Azure DevOps Server supports integration with GitHub Enterprise Server repositories.

Getting Started

When you sign up for Azure DevOps, you must first register an organization, project and version control type. DevOps teams can create multiple projects and differentiate them with separate names and access permissions. Projects can be set to public or private. Teams can also use Azure Boards for collaboration and project management, and can also configure use of Agile practices. After setting up your team, you can use the platform to begin deploying your applications to Azure.

In addition to a collaborative environment, Azure DevOps provides access to an Azure Pipeline service that supports multiple languages and platforms. The Azure DevOps pipeline is a set of services that are grouped into individual projects.

Additionally, Azure Test Plans feature facilitates automated testing. It is also possible to set up deployment pipelines with different components for multiple applications.

Azure Pipeline can track work status, manage backlogs and create custom reports. With the help of the Azure DevOps pipeline, developers can create, manage and deploy their projects in a single environment. With this solution, DevOps teams will have a single point of contact and can communicate with each other.
March 18, 2022