devopsy.kinsta.cloud

Tag: secure software development

Establishing Visibility and Governance for Your Software Supply Chain

Among the top cybersecurity projects for CSOs in 2025 is “establishing asset visibility and strong cloud governance.” But what does this mean in practice? Many organizations have a good handle on their data storage and compute infrastructure. However, while they may know which virtual servers or serverless platforms are in use, they often don’t know what’s in the applications that are running. The Log4Shell vulnerability was a wakeup call: companies scrambled — and often failed — to find all affected systems.

Why it Matters

The pace and sophistication of attacks continue to increase. The model of a lone hoodie-wearing hacker sitting in a dark room defacing websites doesn’t represent the modern threat landscape. Attackers are large, organized operations, often crime rings or even nation-states. They’re not just going after the easy targets; they’re going after impactful targets: companies that can afford to pay large ransoms, critical infrastructure controls, and government databases. Sometimes the motivation is purely financial, sometimes it’s geopolitical. Either way, attackers are well-resourced and determined.

Even if your organization isn’t the direct target, you can fall victim to attackers. A supply chain attack designed to gain access to a bank, for example, could also poison your supply chain. The attackers will gladly take your customer information or hold your servers hostage to ransomware.

Modern software supply chains are incredibly complex webs of third-party code. To properly secure the supply chain, organizations must first gain visibility into all of the components that go into their applications. This is necessary not just on a per-application basis, but across the entire portfolio. From there, organizations can make prioritization decisions based on context and set automated policy enforcement to ensure that applications are securely developed and deployed.

Security requires constant vigilance, and vigilance requires visibility. How can you hope to secure what you can’t see? That means you need to know what goes into your software and you need to know where that software is running. The faster you can start fixing vulnerabilities instead of looking for them, the quicker you’re protected against the latest attack.

Challenges in Securing Your Supply Chain

Most of your software isn’t yours. In other words, your employees didn’t write it; they brought in third-party dependencies. Whether from vendors or open source projects, these libraries provide a foundation that allows your developers to focus on the differentiated functionality that your company provides, which speeds your innovation and delivery.

The tradeoff for this increased speed is taking on the risk of your dependencies’ security posture. Do you know what vulnerabilities exist in your dependencies? If you do, you can’t necessarily fix them yourself. Even if you have a good relationship with the providers of your direct dependencies, they have their own dependencies, and these transitive dependencies can number in the hundreds and be several layers deep. How do you track all of those dependencies for vulnerabilities?

Once you know what vulnerabilities exist in your supply chain, do you know where the vulnerable versions are deployed? 10 days after the Log4Shell vulnerability was announced, less than half of the publicly-reachable vulnerable applications had been patched. That’s a big window for attackers.

Knowing where all of the vulnerable applications are running doesn’t give you the full picture. Just because an application contains a vulnerability doesn’t mean the vulnerable part of the code is used. It may be the case that your application doesn’t call the vulnerable code path or that the vulnerability is conditional on a configuration that you’re not using. While you may still want to address the vulnerability, it’s not as high a priority if it doesn’t affect your current application portfolio.

Establishing Visibility and Governance

Thankfully, you can overcome all of the challenges. The first step is to start building software bills of materials (SBOMs) at build time. The SBOM records what goes into your software, so it is the foundational piece of building asset visibility. You can then use that information to build a knowledge graph about your supply chain, including vulnerabilities and software licenses. When you aggregate these SBOMs across all of your application portfolio, you get a holistic view of all dependencies.

This portfolio-wide view helps you prioritize which issues to fix first. The impact of fixing a vulnerability in one or two applications is a lot less than fixing a vulnerability in all applications. Similarly, fixing a vulnerability in a production, external-facing application is more impactful than fixing a vulnerability in an internal development environment.

As mentioned above, knowing if a vulnerability even affects the application that contains it is a key part of the overall visibility picture. Ingesting VEX (Vulnerability Exploitability eXchange) statements, when available from your dependencies, will help you eliminate the vulnerabilities that are irrelevant to your code. Runtime analysis, like with eBPF, adds more filtering by determining which code paths are actually loaded into memory. This gives you more detail on which vulnerabilities are exploitable in your specific application, especially when VEX statements aren’t available.

Context-aware prioritization makes responding to issues easier and faster. But what’s even better is preventing issues in the first place. Creating SBOMs at build time means you can enforce policies to prevent issues. For example, you can use the SBOM that you just produced in the build pipeline to check for known vulnerabilities in the dependencies and block merging the code into the main branch if it contains vulnerabilities or if it has a poor security posture. After build, tools like Kubernetes Gatekeeper can automatically enforce policies to block deployment of applications with critical vulnerabilities, missing or incorrect cryptographic signatures, and other security concerns.

One final piece of the puzzle is tracking software provenance. Tracking and gating on software provenance gives you another avenue to protect yourself from vulnerable code. This is often overlooked, but given the prevalence of attacks against open source library repositories, it’s more important than ever. Provenance is a provable record of how software was built, where it was built, and by whom it was built. These cryptographically-signed attestations mean that you can know if the package you downloaded from npm came from your dependency’s official build system or from hackerman9000’s laptop after he compromised the project’s npm account. But provenance doesn’t just help with deliberately altered, it also gives you a sense of the security practices used in the build process. Standards like SLSA give you assurance that builds took place in environments with sound security.

Start Somewhere

Implementing all these practices takes work and cultural change. You don’t have to do everything at once; each step you take reduces risk. For example, we know that SBOMs alone won’t solve supply chain issues, but having them is better than not having them. Once you have SBOMs, you can build on them to collect more data. Don’t let the perfect be the enemy of the good. After all, security is a process, not a destination.

October 14, 2025
CISA Pushes Steps to Better Secure Software and Product Designs

The country’s top cybersecurity agency is urging developers to take steps to ensure the software they’re building and the products they roll out are secure and protect end users.

The Cybersecurity and Infrastructure Security Agency (CISA) this week rolled out a series of recommendations that it wants businesses and critical infrastructure organizations to adopt to protect themselves against the growing cyberthreats they face.

The 18 “voluntary practices with high-impact security actions” – divided between recommendations for software development processes and product design – are part of a larger push by CISA over the past few years to secure the software supply chain and to shift the responsibility of security from the users of products to the organizations that create them.

They dovetail with the agency’s secure-by-design efforts, which advocate for address security throughout the software development lifecycle. They also reflect the Secure Software Development Framework developed by the National Institute of Standards and Technology (NIST), according to Chris Hughes, chief security advisor at software supply chain security firm Endor Labs and a CISA Cyber Innovation Fellow.

“They’re good reminders and solid cyber hygiene recommendations that most organizations should be doing, especially those in IT- and product-centric development environments, with ramifications for downstream customers and consumers,” Hughes said.

MFA, SBOMs and More

None of the recommendations should come as a surprise to developers. Regarding software development, CISA recommends developers separate all the environments in the process, such as development, build, test, and distribution, enforce multifactor authentication (MFA) across all of them, eliminate the use of insecure storage and transmission of plaintext credentials, used automated tools to test software for vulnerabilities and deal with security flaws before releasing the software.

In addition, the agency wants organizations to make software bills-of-material (SBOMs) – a key security step that CISA has been advocating for to detail all the components used to create the product – available to customers.

For product design, key steps include broader use of MFA for customers, eliminating default passwords, releasing security patches broadly and promptly, and enabling users to understand, monitor, and respond to cybersecurity issues affecting products, such as initial access, privilege escalation and lateral movement.

Businesses also need to be clear with users when products are reaching end-of-life and security patches will no longer be available for them.

A Good Start

While applauding CISA’s efforts, Hughes said there are other areas that should have been emphasized more.

“There is some language around vulnerabilities,” he said. “It could be taken a step further with an emphasis on prioritizing known exploitation, exploitation probability, and reachability for not just internal vulnerability management efforts but in external communications through vulnerability disclosures, for example.”

Other areas of emphasis could have been open-source software governance and security and broader specifications for cybersecurity supply chain risk management (C-SCRM) for third parties, including software-as-a-service (SaaS) providers.

What About C-SCRM?

Supply chain risk management is a fast-growing market, with Introspective Market Research analysts expecting the space to expand from $3.46 billion in 2023 to $7.68 billion by 2032. C-SCRM is a more specific area of the market that focuses on identifying, analyzing, and mitigating vulnerabilities, data exposures, and similar security issues in IT and operational technology products and services. It’s a process pushed by NIST, the General Services Administration (GSA), and other government agencies.

What will be most challenging is that these steps are voluntary, Hughes said, adding that they “will conflict with competing priorities organizations have, such as speed to market, revenue, feature velocity and market share, all of which CISA has themselves cited as challenges with efforts such as Secure-by-Design.”

It’s also unclear how CISA’s mandate will change under the incoming presidential administration, which has long been an opponent of regulations on the private sector and is expected to narrow the agency’s responsibilities.

January 10, 2025
Can Regulatory Mandates Secure Software Development?

Software companies have a long history of delivering incomplete and insecure products. This happens for a couple of reasons. Fast time-to-market has always been a business priority, taking precedence over security, especially as DevOps has become the norm in software development methodologies. And software, by nature, is easy to update and change, so some defects or oversights could be allowed to slide through with the thought that “We can always fix it later.”

Unlike car manufacturers that put a premium on including safety features upfront (or face the heavy hand of regulators), software makers have typically emphasized speed to market as a way to stay competitive.

But the other shoe often drops on that approach in the form of costly recalls, extensive patch programs, cybersecurity breaches, and other issues. The fix-it-later mentality in software has become increasingly challenging in recent years with the emergence of connected/IoT devices that are harder to keep current and have a greater impact. Connected closed-circuit surveillance cameras, for example, are frequently hunted and exploited on the internet to spy on people. Internet of things (IoT) devices operating at dams or nuclear power plants can now be hacked (as seen at the Bowman Avenue Dam in New York, or three utility companies in Ukraine hit by BlackEnergy malware). And, across the board, the cost of data breaches continues to rise.

A White House executive order (EO) from 2021 is also putting pressure on software manufacturers, DevOps and DevSecOps professionals to improve quality, with a mandate for U.S. federal agencies to enforce secure software development through their contracts and acquisitions as part of efforts to secure the software supply chain. The core difference between this effort versus previous attempts such as FISMA, is the visibility into the software development process.

The attention being paid to secure software development is a step in the right direction, but ultimately it’s still up to the industry rather than regulatory bodies, to change the mindset toward building secure software from the ground up.

Technology is Faster Than Mandates

The EO represents a shift in the landscape, mandating not just secure products, but secure development practices. Although it applies officially to federal civilian agencies and their contractors, federal regulations and other government mandates often become global standards, as has happened with the General Data Protection Regulation (GDPR), the EU’s privacy and security law.

But regulatory mandates aren’t enough. They often move too slowly to be a real solution, especially in a software development environment that is moving steadily towards more agile, fast-paced environments where continuous integration and continuous delivery (CI/CD) is the norm. The timelines for new regulations typically require a two-year window for implementation; if a company already has an applicable product in use, it could be up to five years before they’re required to be in full compliance, such as in the payment industry. By that time, the key elements of the industry best practices have already changed. Simply put, technology moves too fast for regulatory bodies to keep up.

What’s more, the regulations themselves could ultimately fall short if they are perceived to have a high cost burden of compliance. This is specifically true if the technology used in the software development life cycle can’t keep pace with the needs of compliance. In support of the EO, the National Institute of Standards and Technology (NIST) has released Draft Special Publication (SP) 800-218, and, as is routine, accepted open public comments. The feedback from industry consortiums in these situations often includes considerations about the costs of implementing new regulations and can sometimes lead to softening of some aspects, arguing, for example, that the new regulations put an unfair burden on small and mid-sized companies. The final set of regulations can sometimes leave the door open for ways to address compliance without addressing the issues that led to the regulation in the first place.

For example, the California Consumer Privacy Act (CCPA) is widely hailed as the nation’s strictest set of privacy protections, yet it still has its weaknesses. The most commonly cited weakness is that the law makes it extremely difficult for consumers to take advantage of the new regulations when they feel they have been wronged.

Industry Must Make it Happen

Ultimately, it is up to software manufacturers and the industry to improve the security of their software. They should adopt best practices to build secure procedures into the design, development, testing and validation of new software, as well as its deployment and use. NIST’s Secure Software Development Framework is one example of guidance, being drawn from recommendations from groups like BSA, OWASP, and SAFECode. Industry groups, such as the International Organization for Standardization (ISO), could adopt standards for development and certification similar to the universally accepted UL standards for electrical products.

The rising demand for secure development requires a substantial shift in attitudes toward how software is made. A lot of these things can be more easily implemented if organizations invest in the adaptation of new technologies faster.

The threat landscape has become too sophisticated and pernicious for the problem of insecure software to be ignored. Fix-it-later is no longer a valid option if organizations want to protect their networks and, especially, their data, which has become the lifeblood of operations in every sector. While regulatory mandates help, the job ultimately falls to the software industry itself and its consortium of like-minded people bridging the gap between developers and DevOps teams of competing companies to ultimately deliver what’s best for the consumers.

February 23, 2022
Embracing Security by Design in Development

The number of data breaches publicly reported in 2021 far exceeded the total for 2020, making 2021 a record year. With cyberattacks on the rise and bad actors and ransomware gangs demanding millions in ransom, cybersecurity has taken on new urgency.

In addition to more sophisticated hacking techniques, the rapid pace of tech adoption during the pandemic is another contributing factor fueling cybersecurity risks. A recent survey by Forrester shows 74% of companies attribute recent cyberattacks to vulnerabilities in technology put in place during the pandemic. As companies raced to enable remote work, develop new applications and roll out new consumer-facing digital solutions, cybercriminals saw new opportunities to identify vulnerabilities and to attack.

Cybersecurity threats can affect every business regardless of size and every individual, regardless of who they are or where they live. Even the most protected and advanced organizations are at risk: One of the world’s biggest cybersecurity companies recently fell victim to one of the largest breaches in history. It’s time for all companies to rethink their approach to boosting security in the software development process.

As a nearshore development partner for businesses around the world, we have embraced a ‘security by design’ mindset to keep our clients’ builds secure. To put security first and ensure it remains at the forefront of every engagement, we adopted a parallel build systems process to improve the integrity of the code we deliver. This also reduced the overall threat of potential attacks. How did we do this? There are three key components to developing and implementing a security-by-design mindset.

Embedded Security Expertise

It is critical that security experts become official members of every software development team. These specialists are there every step of the way from design to delivery, finding and fixing security flaws along the development journey. Instead of being brought in at the end of the process, they are deciding on new solutions and implementing protections in real-time alongside experts in design, user experience and coding. Each plays a role and works as an integrated member of the team providing advice and suggestions during Agile cycles and sprints.

Predictive Analytics

A recent survey showed that 96% of third-party applications deployed in cloud infrastructure contain known vulnerabilities. That rising risk can make it possible for bad actors to infiltrate thousands of organizations. Threat hunting tools can spot, anticipate and predict vulnerabilities in software development. Tools can automatically scan code as it is written to vet and diagnose vulnerabilities. Our team scans code while in production and reports risks and vulnerabilities and then immediately consulting with the rest of the team to work on the next secure version.

Continuous Learning

Security is changing quickly, and it’s never been more critical to ensure developers have the skills necessary to defend and protect corporate infrastructure, applications and digital solutions. With the threat landscape constantly evolving, new skills and continuous learning are required to stay ahead of the latest threats and fraud techniques. Part of our training is focused on deploying white hats, or ethical hackers, to breach our development environments and the applications we deploy. These experts then set about identifying weaknesses and helping to improve security. They also take our training to the next level and push our experts to continually learn and educate themselves on the latest threats and solutions. By upgrading their skills, security practitioners can stay ahead of cybercriminals.

Every day, enterprises face thousands of attacks on their systems, and malware continues to grow in frequency, volume and complexity. These attacks are not just coming from criminals looking to make a buck; there are growing fears of nation-state attacks escalating. These are often seen as the more dangerous threats to businesses. The best defense against cyberthreats is a strong offense anchored by a security by design mindset.

January 24, 2022
Prioritizing Scalability, Reliability and Security in Engineering
As digital products and services are more deeply embedded in critical industries and infrastructure and the implications of problems grow in scope, engineering organizations are renewing their focus on building platforms that are scalable, reliable and secure. More importantly, they are recognizing that this isn’t the responsibility of an SRE or CSO, it’s an ethos that must be carried each day by every engineer on the team. This is an important shift—from teams and cultures that valued speed of delivery above all else to teams and cultures that make high-quality systems the foundation of everything they do.

This isn’t easy, and big cultural changes don’t happen overnight. There are many things that can go wrong when the team is operating in the legacy mode of prioritizing quick time-to-market but there are also some strategies and best practices for beginning the process of cultural change across your engineering organization.

The Old Way of Doing Things

You’re the engineer on call, and you get woken up at 3:00 a.m. by an alert that tells you an app is down. Groggy and annoyed, you look at the message and have no idea what service it’s referring to or where to begin.

After nearly an hour, you can’t figure out who owns the service or how to get it up and running. So, you call that senior engineer on your team—you know the one—the person who gets called every time there’s a tricky question or problem. Thankfully, they answer your call in the middle of the night and help you bring the app back up. But in a month, the same scenario happens again; only this time, when you go to call that senior engineer, they’re no longer there. They quit to go work at a company that cares about reliability.

Best Practices for Today

How can you avoid this vicious cycle? Here are some best practices you can implement to begin shifting focus away from speed and toward scalability, reliability and security.

1. Evangelize the Narrative That the Systems you Work on Matter

At the end of the day, your systems are there to serve your customers and users. Leadership and teammates who embody these values are incredibly important. People should worry about letting their customers down, not about being yelled at by their manager or receiving a bad performance review.

2. Set Explicit Expectations about Uptime

When you say that your app needs to have 99.99% uptime, it doesn’t have the same impact as saying 52.6 minutes of downtime a year. Translate uptime numbers so they’re tangible—and on people’s minds every time there’s an outage.

3. Don’t Sweep Problems Under the Rug

After the dust settles from an incident, it’s important to get together and talk about it. Be transparent about what happened, which person or team made the error and what the implications to users were. The idea isn’t to point fingers or assign blame but to bring the team together so that you can learn, evolve and figure out how to prevent this from happening in the future.

Here are some questions you can use:
1. Who wrote the code?
2. How was it tested?
3. Should our automated testing have caught this?
4. Who reviewed this code?
5. Were they the right people to review the code?
6. Why wasn’t it caught in the staging environment?
7. Who released it and who went through the release validation checklist?
Formalizing this process lets everyone on the team know that mistakes are part of the job and that there’s a standard way that problems are handled. This is a great way to help engineers feel free to do their best work without fear or avoidance.

4. Define Individual Owners for Every Piece of the System

If there’s any confusion about who owns what within your engineering org, you need to address that right away; it’s a massive problem. Not only do you need an owner for every part of the system, but they must also be empowered to fix issues and know that they’re the ones on the hook.

5. Invest in Documentation, Runbooks and Logs

The owner of each piece of the application is also responsible for making sure that documentation is up to date, runbooks are clear and easy to follow and logs are readily accessible. When you’re getting paged at 3:00 a.m. about a service you’ve never heard of before, these resources are key.

6. Prioritize the Delivery Pipeline Like it’s a Key Feature

You should always be thinking about how to build safeguards around your workflows, like automated testing or new tools. This may mean you need to go toe-to-toe with product managers to make sure the work required to build a robust delivery system isn’t getting kicked down to the backlog.

7. Get Business Buy-In by Using Terms They Understand

Unreliable code is technical debt. And debt is something all leaders understand, regardless of their technical abilities. Stripe published a survey in 2018 that found the average developer spends more than 17 hours a week dealing with maintenance issues, costing $300 billion per year globally. If you’re pushing to invest in SRE practices, those numbers might help you make your case.

Culture comes first, but the right tools can reinforce the culture and drive accountability over time. Tools that help teams keep track of critical information such as service owners, documentation and runbooks can help tremendously. Additionally, having easily available dependency maps of the architecture are key to quickly understanding complex systems. Lastly, setting organization-wide standards for best practices and providing templates for spinning up new services that follow these can help ensure that these things are not forgotten.

This article was based on a talk given by the author at Conf42: SRE 2021.
November 12, 2021
Tips for a Successful DevSecOps Life Cycle

A DevOps implementation, if done correctly, can do wonders for any organization that’s on the hunt for efficiency, productivity and speed. As per the 2020 survey conducted by Atlassian, 99% of survey respondents said that DevOps has had a positive impact on their organization.

However, it won’t do any good if security isn’t prioritized while adopting the DevOps best practices. That’s where DevSecOps comes into play.

Enterprises need to understand that leveraging the alignment of development and operations for workflow enhancement while ignoring underlying security issues is like trying to generate electricity from a broken solar cell on a cloudy night. Undoubtedly, the success of any DevOps initiative predominantly relies on general security practices and a robust security mechanism to mitigate risks—this is where DevSecOps plays a vital role.

There are several essential components of a successful DevSecOps implementation. Here’s how you can leverage them in your organization.

The Emerging Need for DevSecOps

DevOps can be simply defined as the next evolution of an agile development environment that bridges the gap between operations and development teams. Enterprises increasingly are turning to DevOps to boost their application development speeds and operational efficiency. Often, though, prioritizing development velocity means security concerns are left behind.

That’s why there’s an equally growing need for a secure, efficient yet still agile development environment. DevSecOps puts security at the forefront of the entire operational and development process, which safeguards good cybersecurity hygiene right from the beginning. Implementing DevSecOps directly benefits developers and operations teams working hard to deliver a flawless application in a short period of time. Moreover, this approach encourages enterprises to develop secure code and thus, secure applications. To do so, there are several resources and strategies to help implement DevSecOps into an organization’s software development life cycle (SDLC).

Secure Coding Practices

Secure coding is perhaps the most crucial aspect of ensuring stringent security right from the beginning of the SDLC. Developers who aren’t following secure coding best practices, security guidelines and maintaining compliance while coding the application are inviting many security risks, including breaches of sensitive data and exploitation of an individual’s identity.

Hence, an organization must focus on enhancing the skills of developers and ensuring that they are following security best practices while their application is in the initial stages of development. Moreover, establishing and adherence to coding standards are also crucial, as they help application developers to write clean code.

Leveraging Automation

As we know, automation is a driving force and a key aspect in DevOps; the same goes for DevSecOps. Automating security is vital to keep up with the pace of application code delivery, especially for more prominent organizations working on various code versions. Choosing the right automation testing tools can be beneficial for organizations that wish to speed up their operations and enhance the reliability of code when it comes to security. Organizations should strongly consider using at least static application security testing (SAST) tools that help developers identify potential vulnerabilities in code in the early stages of the development life cycle.

Early Stage Testing

The biggest mistake for any organization leveraging DevSecOps is testing the application only when it’s finally completed. Beginning early with automated testing offers numerous advantages, including identifying and resolving vulnerabilities in early development stages, a quick and inexpensive way to fix issues and saving valuable time during deployment.

However, testing during the early stages of the SDLC may also halt the overall DevOps development pipeline, so take that into consideration from a security perspective.

Implementing DevSecOps

Implementing DevSecOps isn’t a quick process and requires specific steps to get it right.

Planning and Development

A solid planning strategy for successful implementation must include test criteria, design, vulnerability models and analysis. Once the planning is complete, the next step is development. Development teams should initially evaluate the overall complexity and challenges of their current practices. Adding a code review system in the development process can help to encourage safe and reliable coding practices in a DevSecOps environment.

Build and Test

Now it’s time to build! You can use automated tools to help combine source code with machine code; many of those build automation tools offer security features like automatic vulnerability detection. Once the build is complete, rigorous testing is carried out through various automated testing frameworks that ensure the highest level of performance and security.

Monitoring

Monitoring is another crucial aspect of DevSecOps implementation to ensure your security mechanisms are performing up to snuff.

Undoubtedly, DevSecOps has revolutionized the way enterprises handle security, especially in challenging DevOps environments. Businesses that want to get to the next level of agility without sacrificing robust security should be implementing DevSecOps.

October 4, 2021
JFrog Acquires Vdoo to Advance DevSecOps

JFrog today announced it has agreed to acquire Vdoo for $300 million in cash to gain a set of analytics tools that discover vulnerabilities in application binaries.

Vdoo’s scanning tools, infused with machine learning algorithms, will be fully integrated with the JFrog Xray vulnerability detection tools along with the rest of the JFrog continuous integration/continuous delivery (CI/CD) platform in 2022. In the meantime, the Vdoo scanning tool will continue to be made available via a software-as-a-service (SaaS) platform that Vdoo built.

The Vdoo platform is already integrated with the JFrog Artifactory repository and JFrog Pipelines. Vdoo’s platform also is integrated with DockerHub, Jenkins, GitHub, GitLab and Azure Pipelines via REST application programming interfaces (APIs) that make the Vdoo scanning tools accessible via a command line interface (CLI).

JFrog CTO Yoav Landman said acquiring Vdoo is critical because when it comes to implementing DevSecOps best practices, the only meaningful place to discover and remediate vulnerabilities is in the binaries deployed in production environments. As such, the Vdoo platform provides a superior alternative to both static application security testing (SAST) and dynamic application security testing (DAST) tools that scan for vulnerabilities in source code, added Landman.

Developers require accurate intelligence that can be quickly acted upon to secure applications running in production. They can quickly take action once they know how a specific vulnerability impacts a binary, versus being told to update source code that could take days or weeks to complete, noted Landman. The Vdoo platform also makes it possible to leverage machine learning algorithms to detect zero-day vulnerabilities, malware, exploits, backdoors, supply chain risks and other threats before they become commonly known. Those vulnerabilities can be detected in everything from application binaries to firmware running on embedded devices. Vdoo is also recognized by the Mitre Corp. that oversees the Common Vulnerabilities and Exposures (CVE) database as a CVE Numbering Authority (CNA) for discovering vulnerabilities.

As responsibility for application security continues to shift left toward developers, many organizations are finding the tools that cybersecurity teams employed to discover threats don’t lend themselves to the workflows that developers have created to build applications. As a result, many of these tools are being replaced by developer-friendly alternatives that developers can invoke via a CLI as part of DevOps workflow. The goal, now, needs to be reducing the level of noise created by security tools to enable developers to focus on the vulnerabilities that specifically impact their code, noted Landman. Otherwise, developers are simply overwhelmed by a massive number of alerts generated by security tools that often don’t appear to be especially relevant, Landman added.

It’s not clear how quickly responsibility for application security is shifting left. However, in the wake of a series of high-profile software supply chain breaches, the urgency surrounding adoption of DevSecOps best practices has increased considerably. The challenge, as always, is getting the right tools into the hands of developers at the right time.

June 29, 2021
Prevent False Positives From Derailing Shift Left
Static application security testing (SAST) tools are designed to balance false positives (incorrect warnings) with false negatives (missed vulnerabilities) primarily because deeper analysis requires more time and computing resources. Both of these are in short supply among developers that are tasked with meeting shorter and shorter product delivery milestones.

So, while SAST vendors consider a true positive a correct detection of a real defect, and a false positive an alert on a bug that does not exist, this is rarely the lens used by developers. What really matters to them is whether the output of a SAST tool is useful and actionable. For example, there is a great deal of variation in how developers interpret SAST scan results, depending on the nature of the defect, the role of the user, the platform on which the application will run and the environment in which it is deployed.

Take, for example, a true positive result of a buffer overrun, one of the most notorious classes of C/C++ defects from a security perspective. In the early stages of application development, it almost always makes sense to change the code to fix such a bug. Developers are actively changing the code, anyway, so fixing it involves little extra overhead. However, if the same defect is found after the application has been deployed, then it is much trickier to decide whether it is worth fixing.

That’s because the cost of fixing a real bug may exceed the benefit of fixing it, and the benefit of “correcting” a false positive may exceed the cost of leaving it alone. Since SAST tools can only be relied on to give narrow technical answers, humans must interpret and determine which static analysis results should be acted upon.

Consider a hypothetical comparison of different SAST tools in the graph below. Tool A has good recall (i.e., the ability to identify real defects) and precision (i.e., the ability to exclude false positives) which results in finding many of the real bugs with a reasonable amount of false positives. Tool B has high precision but poor recall, resulting in low false positives but a higher number of false negatives (undetected security vulnerabilities). Tool C has poor precision but high recall, resulting in detecting all the possible errors but also with a very high number of false positives.

A comparison of recall and precision of hypothetical SAST tools.

Developers hate false positives because they introduce unnecessary work and delays. This, in turn, has a disproportionate effect on the way tools are designed, configured and used. If given a choice between Tool A that reports 40 real defects and 10 false positives, and Tool B that reports 50 real bugs but with 50 false positives, users will almost always prefer the former, even though it is finding fewer real defects. This is perfectly understandable — users are asked to weigh an immediate concrete negative (time spent looking at false positives) against an intangible potential future positive (vulnerabilities that may not show up).

However, if one offsets the time and risk saved in finding those 10 bugs earlier (i.e. by avoiding expensive and potentially dangerous bugs in finished products) against the time needed to assess the additional 40 warnings as false positives, then it quickly becomes apparent that configuration B is more economical. This is especially true with security vulnerabilities that slip through testing and can have expensive consequences.

Since the number of warnings within a DevOps pipeline can be a deterrent to getting the most out of a SAST tool – especially early in the project, when it provides the greatest benefits – organizations should consider the following techniques to increase developer attention to alerts:
- Filter and Focus: Parse the viewed data from the tool’s interface, focusing on what is most important for the project and assigning developers to fix critical issues in priority order.
- Mark and Defer: Lower the priority, or change the state to “later,” for example, on all or a subset of the warnings based on some set of conditions that are less crucial to the project.
- Stop the Bleeding: Use the above techniques to temporarily defer existing warnings with the emphasis on fixing new defects that are introduced as code is changed or new code is developed.
Despite the fact that it will invariably produce some false positives, SAST helps shift security further to the left by ensuring security and programming vulnerabilities are eliminated when code is being written, rather than at the testing stage when they are much more expensive to fix. Filtering and prioritizing warnings are two effective techniques to ensure developers don’t throw out the baby with the bathwater.
May 19, 2021
GitGuardian Reports Careless Handling of Application Secrets

A new report, the 2021 State of Secrets Sprawl on GitHub, published today by GitGuardian, a provider of a tool for monitoring usage of application secrets, suggests developers are not especially good at keeping those secrets safe.

Based on an analysis of every single commit made to GitHub, the report finds there has been a 20% year-over-year increase in the number of secrets – such as application programming interface (API) keys, private keys, certificates, usernames and passwords – discovered on a public repository.

Developers, as they build an application, make secrets easily accessible as plain text for the sake of convenience. However, when those applications are deployed in a production environment, developers often forget to delete the secrets they stored in GitHub.

GitGuardian CEO Jeremey Thomas noted that cybercriminals are now actively scanning repositories looking for secrets that would enable them to compromise applications.

As the number of application development projects being launched increased, so too did the number of application secrets being stored in repositories like GitHub. The GitGuardian report notes 60 million repositories were created in the last year, representing a 35% increase over the previous year. GitGuardian claims it detected more than 5,000 application secrets per day. Most of those secrets have been detected in repositories that belong to developers (85%), while the other 15% belonged to organizations. The top four countries where secrets were disclosed were India, Brazil, the United States and Nigeria.

GitGuardian also notes it sent out 937,539 pro bono alerts to 558,086 developers last year that informed developers that application secrets were exposed. The mean time for GitGuardian’s algorithm to detect a leak is four seconds. The average mean time to response of a developer when informed of the issues is about 25 minutes, the report noted.

As a best practice, GitGuardian recommends not storing unencrypted secrets in git repositories, and that unencrypted secrets should never be shared via messaging systems like Slack. DevOps teams should also restrict API access and permissions.

As more attention is paid to how software supply chains are managed in the wake of some recent, high-profile breaches, there is no doubt the number of audits conducted will increase. Many of those auditors will specifically be trying to ascertain how application secrets are being managed. Developers may have to get used to a new, higher level of scrutiny, especially as DevSecOps practices become more widely adopted.

In the meantime, DevOps teams may want to review their dependency on public repositories. There are plenty of private repository options that provide most of the same source code sharing capabilities as a public repository. That may mean someone on the DevOps team is required to manage the private repository, but, given the fact that many regulatory bodies will soon be looking to make an example of organizations that don’t secure application secrets, chances are good those costs will be trivial compared to a fine that would be levied.

March 9, 2021
How the SRE Role Is Evolving

In recent years, site reliability engineering (SRE) has garnered much interest. In 2019, LinkedIn listed site reliability engineer as the second most promising job in the United States. Now, in 2021, the role continues to grow and evolve within many organizations.

Initially spearheaded at Google and credited to engineer Ben Treynor, the strategy seeks smarter accountability for application reliability. An SRE team sets service level indicators, makes error budgets for new features and uses tools like application performance monitoring (APM) to visualize performance insights, among other tasks. SREs are quickly becoming a key check to increase business output in response to new digital innovations.

I recently met with AppDynamics Regional CTO Gregg Ostrowski to discuss the emergence of SRE and how the approach is evolving. In short, it appears SREs will only continue to grow in importance across more companies, adopting new tactics like chaos engineering, and diversifying their teams with new domain knowledge in response to increasing technological complexity.

What is an SRE?

Traditionally, organizations employed system administrators, or sysadmins, to maintain operations for large computing systems and services. However, this typically produced a dichotomy: product developers want to push new features, while operations teams (sysadmins) want to make sure the existing service doesn’t break. The split between developers and operations can be unhealthy and cause friction.

Whereas sysadmin positions are detached from development and involve a lot of manual work, an SRE approach, on the other hand, takes a more engineering-focused approach to automate operations. Google’s SRE book, which has become a bible of sorts, defines the role as “What happens when you ask a software engineer to design an operations team.”

So, what does an SRE do? Well, at Google, they are in charge of availability, latency, performance, efficiency, change management, monitoring, emergency response and capacity planning for the services they operate. At Google, they spend no more than 50% of their time on traditional “ops,” like tickets, on-call and manual tasks, and are encouraged to use the remaining 50% of their time to develop automation that will replace human operations labor.

5 Ways the SRE Approach is Evolving

It’s been a few years since Google introduced the SRE concept, and naturally, unique iterations have popped up within other enterprises. But, on the whole, where is the concept heading? Ostrowski sees SREs evolving in a few key areas: growth and maturity, increased diversification with domain-specific experts and new monitoring tactics.

1. Increased Adoption

First, not all companies have embraced an SRE model. A recent study by Blameless found “… 50% of respondents employ an SRE model with dedicated engineers focused on infrastructure and tooling, or an embedded model where full-time SREs are assigned to a service.”

The SRE model is gaining momentum, but there is still room for greater adoption. There is also room for internal growth. Ostrowski sees a single SRE team as a single point of failure. “It needs to be a whole department,” he said.

In addition, SREs are gaining a more prominent voice at the table, influencing feature rollout. “With proper and mature SRE involvement, teams can’t willy-nilly deploy,” he said. Ostrowski views these teams as maintaining a critical balance between business risk and introducing new technology.

2. Larger, Diversified SRE Departments

Many companies are experiencing rising user demands, and thus must rapidly scale their application networks. Simultaneously, there has been a Cambrian explosion of deployment types — systems could be using any assortment of legacy infrastructure, mainframe, microservices, cloud environments and multiple cloud vendors. “The complexity and topology of the IT space has grown substantially, with many interdependencies,” Ostrowski said.

Due to increasingly complex technical stacks, SRE must now cover many domains. Thus, SRE departments will likely require a more diversified team of domain experts. Ostrowski likens SREs to “Navy SEALs of the IT team.” For example, assigning them with experience navigating the nuances behind Google Cloud Platform (GCP) or Microsoft Azure could maintain service reliability in respective clouds.

Individuals’ personality matters for suitability to SRE, too, said Ostrowski. The role requires a specific type of innovator who considers the repercussions of code, and loves improving processes and performances.

3. New Testing Tactics Will Emerge

Ostrowski also foresees SRE departments introducing new monitoring and testing approaches to maintain reliability. One of these is chaos engineering, which champions the idea of intentionally breaking application systems. Different strategies will undoubtedly emerge to help drive user experience (UX) and ensure that performance is always top of mind.

4. Businesses Rely on SREs to Mitigate Risk

All things will inevitably fail, at some point. SREs accept failure and learn to manage it, designing repeatable operational mitigation structures. “As companies become reliant on the consumerization of IT, business is dependent on SREs to drive business,” said Ostrowski. This reliance will likely increase as businesses tap SREs to maintain stability.

Whether it’s reducing mean time to repair (MTTR), programming service level indicators to monitor website load time, or forecasting error budgets for new feature introductions, SREs will be increasingly relied upon to maintain business stability and high performance.

“The SRE mindset is about coming to terms with a blameless environment,” said Ostrowski. They can help “balance a risk between the business and application team,” described Ostrowski.

5. SREs Steer UX

In this digital-only environment, “The application has become the business,” described Ostrowski. If the application is core to the business, monitoring the user journey is necessary to improve it. Ostrowski believes SREs oversee a unique territory that could produce valuable business insights too.

In addition to monitoring uptime to ensure URL response times meet SLAs, SREs could track UX-related insights, such as conversion rates or cart abandonment percentages. Tracking such analytics and setting standards baselines could help pinpoint problems affecting UX. It could also assist product development in designing better-performing software.

Final Thoughts

“Anything that you do more than twice has to be automated.”
-Adam Stone, CEO, D-Tools

Traditional operations teams are typically detached from product development and are culturally very different. However, within an SRE approach, operations instead aim to run systems that automate work typically performed manually by sysadmins.

In this brief introduction, we barely scratched the surface of the topic. And of course, each company is unique in its approach to hybridizing product development and operations. DevOps tenants are closely aligned with SRE — often they are intermixed or one and the same.

Ultimately, truly reaping business value from this approach will depend on breaking down silos and opening conversations — or at least automated notifications — between disparate units. Only then can operations and development coexist in the most productive way.

February 10, 2021