devopsy.kinsta.cloud

Tag: devsecops

Business Leaders Will Trade Speed for Security

A global survey of 600 C-level executives conducted by CloudBees found that when it comes to building software, more than three-quarters of respondents said it is more important to be secure and compliant than fast and compliant.

As a result, more than three-quarters (77%) also noted their organization is implementing a shift left strategy for implementing security and compliance even though 58% recognized that those efforts created additional burdens for developers.

Three-quarters of respondents also conceded that compliance (76%) and security (75%) challenges limited their company’s ability to innovate. Nevertheless, 83% said shift left is important for them as an organization despite the fact that 88% of executives said their software supply chain is secure or very secure. However, only 33% of respondents said their software supply chain is completely compliant.

Not surprisingly, 86% are focusing on compliance more now than they were two years ago and, in the wake of a series of high-profile breaches of software supply chains, 82% are more concerned about attacks.

CloudBees CEO Anuj Kapur said it’s clear there is a greater need to reduce security and compliance burdens for developers by putting guardrails in place that automate DevSecOps processes. The survey noted that only 22% of executives believed their software delivery supply chain is completely automated, while another 37% said it is close to being automated. Only 22% said their compliance process is completely automated, with 35% believing it is almost completely automated.

Three in five (59%) executives said they have all, or mostly all, external tools for security and compliance issues and 29% said they have a mix of internal and external tools. Only 11% use mostly internal tools. A full 90% said their risk management team has the tools, knowledge and expertise to build and/or maintain a secure software supply chain.

The real challenge, of course, is that while compliance issues are relatively static, new security issues arise all the time. Developers make mistakes or a new zero-day vulnerability is discovered after an application has been deployed in a production environment. It’s critical for organizations to embrace automation to enable developers to address those issues as part of a continuously automated update cycle, noted Kapur.

As more responsibility for compliance and security is shifted left, the degree to which DevOps workflows will need to be adjusted will naturally vary by organization. However, the days when developers could ignore security issues during the application development process are all but over. The leadership of organizations of all sizes are making it clear they want to see a reduction in the number of security and compliance issues that arise in production environments.

Ideally, efforts to build more secure and compliant applications would not slow down the rate at which applications are being built and deployed. However, the CloudBees survey made it clear that more executives are willing to prioritize security and compliance over speed of application development. The issue is how to strike a balance between what are clearly two competing priorities.

October 3, 2022
What Developers Need for Software Security Success

Given today’s evolving threat landscape, organizations and businesses in every sector now have a critical need to produce secure software. Criminal gangs, professional attackers and hostile nation-states are employing advanced tactics designed to exploit any vulnerabilities in programs, applications, networks and even raw code. Attackers are constantly finding new ways to bypass even the most advanced protections and defenses. For example, many have shifted their focus from delivering malware to instead compromising credentials, or launching targeted attacks against a supply chain. And while those high-level incursions are happening with much greater frequency, so too are the more basic attacks like ransomware and SQL injections that have been a scourge on cybersecurity defenses for many years. It’s becoming apparent that while cybersecurity platforms and defenses are critical components in defense against modern attacks, what is truly needed is secure code that can be deployed free from vulnerabilities. And that requires security-aware developers with verified security skills.

Most developers say they are willing to champion security and commit to higher standards of code quality and secure output, but they can’t do it without a lot of support, as well as a reworking of the traditional metrics by which they are often judged by their employers and organizations.

Why Most Developers Don’t Prioritize Security

Coding best practices have continued to evolve over the years in response to business needs and market trends. In the past, most applications were created using the so-called waterfall development model where software engineers worked to get their code ready to meet an ongoing series of milestones or goals before moving on to the next phase of development. Waterfall tended to support the development of programs that, having met all of the previous milestones along the way, were free from bugs or operational flaws by the time they were ready for the production environment. But it was slow going, with sometimes 18 months or more between starting a project and getting to the finish line.

The Agile method tended to replace Waterfall, putting a much greater emphasis on speed. And this was followed by DevOps, which is built for even more speed by combining development and operations together to ensure that programs are ready for production almost as soon as they clear the final development tweaks.

Putting speed over security—and nearly everything else beyond functionality—was a necessity as the business environment evolved. In a cloud-based world where everyone is online all the time and mobile transactions by the millions can happen every few seconds, getting software deployed and into the continuous integration and continuous delivery (CI/CD) pipeline as quickly as possible is critical for business success.

It’s not that organizations didn’t care about security. It’s just that in the competitive business environment that exists in most industries, speed was seen as more important. And developers who could match that speed thrived to the point where it became the primary means by which their job performance was judged.

Now that advanced attacks are ramping up so dramatically, deploying vulnerable code is becoming a liability. The preference is once again shifting, with security increasingly becoming the primary focus of software development, with speed a close second. Bolting on security after the fact is not only dangerous, it also slows the process of deploying software. That has led to the rise of programs like DevSecOps that attempt to merge speed and security together to help generate secure code. But developers trained for pure speed can’t become security experts without a lot of help and support from their organizations.

What Developers Need

The good news is that most developers want to see a shift to secure coding and a reprioritizing of security as part of the development process. In a comprehensive survey conducted by Evans Data of over 1,200 professional developers actively working around the world earlier this year, the overwhelming majority said they were supportive of the concept of creating secure code. Most also expected it to become a priority in their organizations. However, only 8% of the respondents said that writing secure code was easy to accomplish. That leaves a lot of room for improvement within most organizations’ development teams between what is needed, and what is required in order to get there.

Simply mandating secure code won’t get the job done. Development teams need training, support and a change in how software engineers are valued and judged within their organizations.

The biggest thing that is needed is more and better training for them. And it should be customized so that less experienced developers can begin their training by learning how to recognize the kinds of common vulnerabilities that often creep into code, with lots of hands-on learning and examples. Meanwhile, more advanced developers who demonstrate their security skills can instead be tasked with things like advanced threat modeling concepts.

Teamwork should also be emphasized so that the developer community can help one another grow their skills. Skilled and willing developers who know security should be appointed as security champions. Their responsibility as a champion will be to help fellow developers improve their skills. And while a security champion is almost always an informal title, they should be given the respect, rewards, and compensation that such an important position deserves.

In addition to funding and supporting training programs and security champions, including giving developers enough time away from coding in order to properly participate in those programs, organizations also need to change the way that developers are evaluated. The primary metric for rewarding developers needs to shift away from raw speed. Instead, evaluations could reward those developers who can create secure code that is free from vulnerabilities or exploits. Yes, speed can be an evaluated factor as well, but first and foremost, code needs to be secure and free from vulnerabilities.

Producing insecure or vulnerable code is no longer an acceptable risk for most businesses. And bolting on security after the fact is becoming increasingly ineffective. Thankfully, the best weapon to fight this disturbing trend is having the developer community produce secure code that attackers can’t exploit. Most developers are willing to step up to that challenge. They just need the proper training and support to help make it happen.

September 29, 2022
What a Dev Wants: Security Training for Today’s Developer Workforce

It’s no secret that developing a strong security culture is a critical component of the production of secure products, yet organizations often overlook the fact that their security training programs set the foundation on which this culture is built. When it comes to developer training, our goal should be to align the intrinsic motivations of a learner with the company’s strategy and objectives. But if either of these is misaligned, the training program, and ultimately the organization, will suffer because it is either irrelevant or perceived as a compulsory box that needs to be checked.

Addressing the Interests of Developers Vs. the Organization

Unfortunately, there is a gap between the buyer persona and the developers themselves, even though developers tend to have a fairly consistent response about the ways they want to be trained. Rather than implement a program that is doomed to be viewed as a time-consuming compliance requisite, organizations should rather ask this simple, yet pertinent question: What do developers really want out of their security training program?

When it comes to today’s security training programs for developers, there’s often a separation between compliance-based training and developer-centric training; in other words, the interests of the organization don’t always align with the interests of the developers. In most highly regulated organizations, there is a mandated security training curriculum that all employees, including the development teams, are required to go through on a yearly basis. This leads to a drawn-out, painful process that leaves employees clicking through modules as fast as possible. Developers may then need to go back and look up security topics on a regular basis if they are looking to implement them into their code.

While developers are certainly making an effort to adhere to security requirements, this greatly slows the ability to get the product to market in a secure and efficient way.

What Training do Developers Really Want?

Organizations should prioritize programs that will specifically target their development teams by asking what those developers are actually looking for. With so many training programs on the market, it can be difficult for an organization to determine which will resonate the most with their employees, specifically their development teams. The fact of the matter is that, when it comes to security training, the majority of developers want to do as little of it as possible. They are willing to check the boxes for compliance purposes, but they want it to be as seamless as possible. As the complexity of training tactics continues to evolve, an ideal curriculum is simple, like programs that can be embedded in the tools developers use on a daily basis.

The most effective training programs are engaging and relevant. Managers tend to see better results from training that is hands-on, interactive, engaging and related to developer tasks. Gamification is also a great way to engage developers. For instance, organizations have started livening up their training curriculum with leaderboards. In fact, recent studies have indicated that developers would opt into shared company leaderboards with points for content consumed. Not only are these programs engaging for the developers, but they are also incredibly effective, as they motivate teams to view more content by leveraging individuals’ pure competitive drive.

Security Champion Programs for the Win

Organizations could also implement a security champions program. These programs vary depending on the organization, but in its simplest form, a security champions program designates a “champion” from each team. The champion is then tasked with leading the team as a subject matter expert as far as security goes. One thing to keep in mind here is that this program could be a major failure if done improperly. Remember, you don’t want to designate just anyone as the team security champion; you want to be sure you’re selecting an engaged employee that will prioritize secure practices and proactively communicate updates across the team.

Regardless of what avenue an organization chooses, the name of the game is ensuring development teams are actively engaged with a relevant and realistic training program. The result of which will have an important downstream impact as organizations work to build a culture with a strong foundation in security. After all, “checking the box” can only go so far when it comes to developing a winning—and secure—solution.

September 20, 2022
Survey Surfaces Massive Number of Application Vulnerabilities

A survey of 16,510 IT and IT security practitioners published today by Rezilion, a provider of a platform for automating the remediation of software vulnerabilities, found nearly half of respondents (47%) worked for organizations that have a backlog of vulnerable applications. More than half (66%) said their backlog consisted of more than 100,000 vulnerabilities and the average number of vulnerabilities that need to be remediated is 1.1 million.

The survey, conducted by the Ponemon Institute, also found more than half of respondents ( 54%) said they can only patch less than 50% of the vulnerabilities.

Factors that keep teams from remediating included an inability to prioritize what needs to be fixed (47%), a lack of effective tools (43%), a lack of resources (38%) and not enough information about risks that would exploit vulnerabilities (45%).

More than a quarter (28%) also said remediation is too time-consuming. The survey found that 77% of respondents said it takes longer than 21 minutes to detect, prioritize and remediate just one vulnerability in production.

In addition, 85% said they spend more than 16 minutes detecting one vulnerability in development. A full 82% also noted it takes longer than 21 minutes to remediate one vulnerability in development. A majority of respondents said it is either very difficult (36%) or difficult (25%) to remediate vulnerabilities in applications.

On the plus side, more than half (56%) said they use automation for vulnerability remediation, with 43% reporting times to respond are significantly shorter as a result.

Rezilion CEO Liran Tancman said the survey makes it clear there’s still plenty of room for improvement. The number of vulnerabilities that need to be addressed is staggering, he said. IT teams need to focus their immediate efforts on identifying the vulnerabilities that are the most critical, he noted.

Remediating those vulnerabilities will soon become a higher priority as more regulations focused on securing software supply chains come into effect. In fact, the software bills of material (SBOMs) that these regulations require will expose much of the dirty laundry that currently exists in application environments, said Tancman.

Ultimately, the goal needs to be enabling organizations to address vulnerability issues without slowing down the rate at which applications are being built and deployed, Tancman added. The degree to which that goal can be achieved will vary widely as more responsibility for security is shifted left toward application developers. However, the need to provide tools to developers that enable them to address vulnerability issues has never been more apparent, noted Tancman.

It’s clear the level of application security vulnerability debt that organizations will need to address can no longer be ignored. The issue now is to determine how best to resolve these issues either by fixing existing applications or replacing them with new ones. Either way, it’s clear that there is a need to adopt DevSecOps best practices to better secure software.

In the meantime, as application security becomes a boardroom-level issue, there be a lot more scrutiny applied all across the software supply chain.

September 14, 2022
DevOps World 2022: Developer and Security Links Protect Your Supply Chain

Ever since the SolarWinds attack back in December 2020, software supply chain attacks have been top-of-mind for any company that builds software. The idea of endangering not just your organization by being attacked but also your customers really sharpens your focus.

Another complicating factor is that applications use a combination of open software components and homegrown code to deliver applications. This software supply chain means your software can be compromised through no fault of your own but instead based on the components you use. The Log4j attack is an example of a component creating vulnerabilities in any software that uses it. So how do we defend our software? We need to think differently about where security starts and how security concepts can be integrated into software earlier in the development process where it interfaces with the rest of the technology infrastructure.

Building a Secure Pipeline

Modern software development is built on the CI/CD pipeline. It allows the application security testing and software composition analysis to happen way earlier in the integration process (shift left), and that allows you to identify and fix issues before the code gets anywhere near customers. Sounds simple, no? Just run your code through a pipeline, integrate security testing, identify issues, fix them, and deploy secure software.

As with most things, what sounds simple in theory can be challenging to implement in reality. First, shifting left requires a collaboration between developers and security teams. Finding the security issues don’t matter if they don’t get fixed. To increase the likelihood of these issues getting fixed, the security team needs to provide context to the developers regarding the urgency of the flaw or vulnerability and some guidance on the best way to fix the issue. To be successful, security folks can’t just drop a report with hundreds of defects and vulnerabilities on the developers and expect that anything productive is going to happen. If this scenario reminds you of getting a report from your vulnerability scanner with hundreds of issues you won’t fix, well, that’s because it’s very similar.

Second, security must add value to the development and architecture process by offering up infrastructure-as-code (IaC) templates and a toolchain that makes it easy to implement secure application stacks. Maybe that looks like a template library managed by a cloud security Center of Excellence, which can provide these tools across all of the application teams leveraging efforts.

Combatting Supply Chain Attacks

Finally, you can implement a security champions program, training a developer (or two) on each team about security issues and they can act as a security emissary to the developers. This provides scale, ensuring that the security team doesn’t become a bottleneck to fixing the security issues.

Given the sophistication of the software, the attack surface of the code, and the velocity of the development process, combatting supply chain attacks requires tight and ongoing collaboration with the development team and shifting left to catch the security issues earlier in the integration process.

Want to learn more about this and other related topics? Join CloudBees for DevOps World 2022, held in Orlando, Florida, at the World Marriott Center. Use discount code DW22 when registering and get access to all the fantastic offerings, including keynotes, sessions, training and other interactive activities. We can’t wait to see you there for the DevOps Remix!

September 9, 2022
Federal Agencies Share DevSecOps Guidelines

The National Security Agency (NSA), Cybersecurity and Infrastructure Security Agency (CISA), and the Office of the Director of National Intelligence (ODNI) have published a set of DevSecOps best practices based on the Enduring Security Framework (ESF).

Developed by a public-private cross-sector working group led by NSA and CISA, the ESF framework was created in the wake of the breach of the SolarWinds software supply chain that led to malware being distributed downstream to a number of organizations that relied on the company’s IT management platform.

The Securing the Software Supply Chain for Developers set of best practices published by NSA, CISA and ODNI takes the ESF a step further. The guidance walks organizations through the processes required to develop secure code, verify third-party components, harden build environments and deliver code.

The best practices provided are the latest dividend of an executive order issued by the Biden administration that requires every federal agency to review the security of its software supply chains. The expectation is that enterprise IT organizations will conduct similar reviews using the best practices and frameworks defined by the federal agencies that specialize in cybersecurity.

The challenge, of course, is embedding those best practices within existing DevOps workflows. In theory, more responsibility for cybersecurity is shifting left toward application development teams. However, the level of cybersecurity expertise that exists today among application developers is limited at best. Cybersecurity was an elective in most developers’ training and education programs, so it should come as no surprise that few of them ever attended a cybersecurity class.

There’s a clear need to improve cybersecurity training for developers, but the real work involves automating cybersecurity processes within the workflows managed by the tools developers employ to write code and the platforms that DevOps teams use to deploy applications, said Mitch Ashley, principal analyst for Techstrong Research, an arm of Techstrong Group that is also the parent company of devopsy.kinsta.cloud. “Security needs to be part of the toolchain, workflows and how the software is designed,” he said.

Ideally, organizations should appoint a security champion that will work with development teams to consistently implement a set of DevSecOps best practices, added Ashley.

Of course, the security champion doesn’t eliminate the need for cybersecurity professionals. Rather, it reduces the pressure when there is a chronic shortage of cybersecurity professionals. In the absence of any formal cybersecurity review during the application development process, it’s simply too easy for developers to inadvertently introduce malware into a software component or misconfigure a cloud infrastructure service. In either case, cybercriminals are increasingly finding it easier to compromise software supply chains at multiple points.

It’s not clear how much the federal government’s guidance might impact the overall state of application security. However, one way or another the level of security scrutiny being applied to how applications are built and delivered is only going to increase in the months ahead. The challenge is finding a way to deliver secure code without necessarily slowing down the rate at which applications are currently being built and deployed.

September 6, 2022
Supply Chain Security: Has the Next SolarWinds Already Happened?

More than two years after the now-infamous hack of the firm SolarWinds, the incident is very much in the foreground of conversations about cybersecurity defense, threat detection and incident response. As evidence of that: The SolarWinds incident was the subject of five separate talks and panels at the recent RSA conference in San Francisco. Dozens more talks addressed issues related to supply chain security that–directly or indirectly–invoked the SolarWinds incident.

That’s because the SolarWinds attack raises important—if not existential—questions for both software development and DevOps teams and the companies running the software. First: If sophisticated cybercriminals can insert malicious backdoors directly into application code from reputable developers, what organization can afford to blindly trust software? Second: If it is now the responsibility of software teams to spot sophisticated, methodical compromises of their development pipelines, what is the best and most reliable means of doing that? Finally, if it is also now the job of software consumers to monitor for—and detect—compromised binaries from otherwise reputable vendors (like SolarWinds), how should they go about doing that given ongoing staffing and technology constraints?

The answers to these questions are matters of debate within DevOps, software development and information security communities. What is beyond debate is that the SolarWinds incident was not a fluke. In the months since that attack, attackers—both criminal and nation-state-backed—have leveraged software supply chain security blindspots to target development organizations as means of gaining access to sensitive IT environments. While we haven’t yet heard of another incident of the scale or importance of SolarWinds, it is entirely reasonable to assume that the next SolarWinds-caliber compromise could already be unfolding, but has yet to be discovered and/or publicly acknowledged.

Software Supply Chain Security Weaknesses

SunBurst, the malicious implant in SolarWinds software, endures as a topic of interest and conversation for information security professionals and for DevOps and development organizations precisely because of the ways it exposed glaring weaknesses in the security of development organizations and software supply chains. As noted in my analysis of the hack in December 2020 and FireEye’s analysis of the same compromise, the SolarWinds hackers gained access to the company’s development environment months before they actually launched their attack. During that time, they closely studied the underlying source code and mimicked SolarWinds style and nomenclature as they added malicious code and integrated it into the SolarWinds Orion application via a compromise of the company’s build system.

Though carefully planned and conducted, the SolarWinds attack wasn’t foolproof. The attackers left clues, such as their additions of native Windows functions, use of cryptography, compression and more. The attackers obfuscated these using compression and Base64 encoding designed to fool simple threat hunting rules that looked for their plaintext variants and other suspicious strings. Still, the obfuscation repeated throughout the implanted code was enough to send up a red flag to anyone who bothered to look closely. That left the attack open to discovery.

The problem is this: Very few software publishers have the staff, expertise or tooling to comb through massive codebases to spot a handful of sketchy obfuscated strings or analyze large binaries for deviant behaviors before deploying them. As evidence of this, ReversingLabs sponsored a survey of more than 300 IT professionals at software-driven enterprises. It found that 87% were aware that software tampering, like what occurred in the SolarWinds attack, could lead to enterprise security breaches. About 40% of those surveyed said CI/CD toolchain exposures posed a risk to their organization. Still, barely a third of respondents (37%) said they had the means to detect such software-tampering attacks.

Threats to Software Development Organizations Grow

Attackers are aware of this weakness within the software development pipeline, which is why threats to development organizations are metastasizing. Just in the last year alone, we have seen a series of attacks on development infrastructure and software supply chains. In one April 2021 incident, attackers were successful in pushing malicious, unreviewed code directly to the PHP main branch, ultimately resulting in a compromised, formal version of PHP being pushed to all PHP websites. That same month, there was a breach at the firm CodeCov, which involved unauthorized access to and modifications of the company’s Bash Uploader script. An investigation of that incident ultimately concluded that the alteration of a single line of code in that script by an unauthorized third party enabled the attackers to (potentially) export information stored in the users’ continuous integration (CI) environments.

More recently, ReversingLabs observed proof-of-concept and “white hat” activities testing so-called “dependency confusion” attacks, in which attackers set up malicious packages in public repositories like GitHub that squat on the names of private, non-public repositories and sport high version numbers. Misconfigurations in package management systems can result in these malicious components being imported in lieu of the legitimate, privately managed namesakes. As ReversingLabs found, a number of malicious modules discovered on the npm platform were used to target German technology, manufacturing and entertainment firms. It was later revealed to be the work of a white hat firm conducting red team exercises on behalf of those firms. Red team exercises aside, RversingLabs research also detected other, malicious packages in npm, as well as similar attacks targeting Ruby Gems and Python Package Index (PyPi).

And that doesn’t even count issues like the Travis CI flaw that exposed (potentially) millions of credentials to developers’ accounts on GitHub, Amazon Web Services and Docker Hub. Lapses like that can fuel supply chain attacks, giving malign actors the ability to insert malicious code into otherwise legitimate open source or proprietary code repositories. After reading such headlines, and without the means to check code intent, how can developers trust their software to update? And how can they do so while at the same time being forced to frequently update their dependencies to reduce or eliminate critical software vulnerabilities?

Dev Teams Need New Tools

For software development organizations struggling to meet delivery deadlines and satisfy customer expectations, reorienting to address amorphous software supply chain threats and attacks is a bridge too far. The same is true for enterprises struggling to staff security teams and simply keep on top of targeted phishing attacks or threats to public-facing IT assets. Too often, these entities only find religion about software supply chain attacks after they have occurred.

What software development organizations need is a way to secure the CI/CD pipelines and release packages, ensuring that their software builds are free of code tampering and other supply chain risks introduced through build systems. That should include the ability to detect compromises by monitoring for malicious software changes and enforcing security controls and failure conditions on software build processes to ensure that you can release your code with confidence.

On the customer end, organizations need a way to detect malicious behaviors and unauthorized software changes in compiled binaries before they are released into production, while also monitoring for risks like exposed secrets or exploitable software vulnerabilities that might lurk in compiled code.

September 2, 2022
The Secret to Secrets Management: Reduce Source Code Vulnerabilities

We are in the cyberattack age. From Microsoft to Crypto.com, no company is completely safe from malicious attackers, and it seems as if a new major cyberattack is announced weekly, if not more frequently. As more companies not only go digital but expand their digital offerings and initiatives, there are more opportunities for cyberattackers. Security teams within IT organizations are hard at work to ensure that their company isn’t next in the headlines. However, as the tech world moves to everything-as-a-service (IaaS, PaaS, SaaS, etc.), the move toward everything-as-code is happening in tandem. As a result, many developers are unknowingly keeping sensitive data (secrets, passwords, certs, keys, etc.) stored in Git source code repositories. This information is often pushed to production which puts sensitive information at greater risk of exposure.

The good news is there is a way to ensure that secrets are locked tight within source code. Let’s explore the complexity and realities of securing source code today and how to take the guesswork out of finding at-risk code that could land your company on the front page of the news tomorrow.

Secrets Management is Challenging and Costly

Sophisticated cyberattackers will stop at nothing to breach valuable data assets. Secrets leaked from source code make it easy for threat actors to access applications using the same credentials intended for legitimate purposes.

At the pace of continuous software development, developers must move quickly to build applications. However, security teams that are expected to prevent breaches are not often well-versed in all facets of the software development life cycle (SDLC). This becomes an issue as the number of lines of code (and the secrets kept within) rise alongside the number of apps developed. According to a recent report from 1Password, “two in three (65%) IT and DevOps employees estimated their company has more than 500 secrets.” And, according to the same report, “the average cost of a secret’s leak is $1.2 million. Poor secrets management can result in organizations losing $8.5 billion annually.” Secrets are vital to building any connected application or process. If hard-coded or left unprotected in source code, the proverbial keys to the kingdom are at risk of exposure—as are your loyal customers.

It’s clear that source code security should be a priority for security teams and DevSecOps professionals. Let’s examine how they can achieve this without sacrificing developer productivity.

How to Reduce Threats Through Secrets Management Best Practices

Manually searching source code repositories for vulnerable secrets can be almost impossible. However, there are some best practices to follow to ensure vulnerabilities are caught early and often:

● Identify and remove secrets in code: You can’t protect what you can’t see. It’s important to have visibility into source code to be able to see where vulnerabilities may exist. There are a few ways of potentially going about shining a light on vulnerabilities. One is through manual scanning of source code, but this takes away valuable time developers could be spending on more innovative tasks. There are also some open source solutions that can help with scanning source code repositories, but as with everything open source, this means developers will have to maintain it (keep it up to date, integrate it, etc.). Instead, security teams should consider looking for a unified toolchain platform to help with source code vulnerability identification and remediation.
● Alert and remediate in code repositories: Provide a fast feedback loop to a developer or DevOps engineer before releasing code into production. Remediation should happen at the time of commit. The idea here is to block secrets from being committed to your source code repositories in the first place.
● Adopt a secrets management tool: Don’t store secrets or binaries within source code—nothing but code should be in your source code repositories. Remove secrets to a secrets manager (i.e. HashiCorp Vault), keys and certificates manager (i.e. Azure Key Vault), etc. Modularize and isolate these managers, but make sure they still communicate with each other so that they don’t expose secrets.

As cyberattacks continue to increase, it is imperative that security teams take a hard look at their source code to ensure that it is locked up tight. Keeping secrets out of source code entirely is recommended in theory but difficult to ensure in practice. Following the above best practices will help reduce the risk of exposure of sensitive data and reduce the risk of losing loyal customers and revenue dollars.

August 30, 2022
GitLab Research Shows Security Concerns in DevOps Teams

The threat of breaches is top-of-mind for many DevOps teams, and while security continues to shift left it appears to be a driving force for many decision-makers when it comes to choosing a DevOps platform or other technologies.

These were among the results of a GitLabs survey that also found getting security right is the number one challenge for DevOps teams.

Getting it Right

Among the top reasons respondents said they chose DevOps were better code quality, developer productivity and operational efficiency—cited by 37% of survey takers, followed very closely by better security/more secure applications.

The survey also revealed that 53% of developers said they are fully responsible for security in their organizations, a 14% jump from 2021.

Michelle McLean, vice president at Salt Security, said it is risky for organizations to depend on development teams alone for security.

“Security and developer teams need to collaborate and work together–at every point in the application life cycle–not just development,” she explained.

For example, security teams must bring runtime insights back to development so they can be applied for education and training.

“It’s fundamentally important to choose a DevOps platform that either has security capabilities built-in or that can easily integrate with security platforms to facilitate collaboration by security and DevOps teams,” she said. “Otherwise, organizations run the risk of pushing out unsecured software or introducing other risks into the software supply chain.”

Tim Mackey, principal security strategist at the Synopsys Cybersecurity Research Center, added that an obsolete mantra in the cloud-native community held that security wasn’t a high priority as proper DevOps workflows meant that incremental changes were small and easily deployed.

He said the Log4j and Log4Shell experiences called that philosophy into question with a recognition that security issues can, and do, occur out-of-band from a release or feature and that continuous monitoring for external threats is part of a properly functioning security process.

“By implementing monitoring for new threats, regardless of source or nature, the ability for cloud-native development to quickly issue tested patches becomes itself a security feature,” he explained.

Understanding the Pipeline

McLean pointed out that while there is a growing understanding of security in the pipeline with DevSecOps, it’s not enough.

“This is particularly true, for example, in the case of APIs. Because APIs aren’t just straight code, you must see them being exercised to spot logic flaws,” she said. “You need the ability to monitor them in runtime in order to spot anomalies and find areas where APIs might be exposing critical data. Although developers write APIs, their testing often falls short for security.”

To accurately see what’s going on with APIs, developers need to exercise the API–even in pre-production.

McLean said this is an area where security and development collaboration and integration is essential–APIs underpin a significant portion of the modern economy and the disconnect between security and development teams can have a direct economic impact.

John Bambenek, principal threat hunter at Netenrich, a security and operations analytics SaaS company, added there has always been a delay between the release of new platforms and the awareness that security capabilities for those platforms need to be implemented.

“With cloud-native and serverless technologies, that delay is as short as I remember in my career,” he said. “I greatly prefer the world where everyone at least understands the need for security instead of having to spend time getting buy-in that it matters in the first place.”

He said if you can manage and implement security in a seamless and efficient way early in the development process, it is far easier and cheaper to address issues than addressing them after code has already shipped—and that’s without adding in breach or liability costs.

“You can either fix it in dev or in prod, but you’re going to have to fix it sooner or later,” he said.

McLean noted many organizations are still maturing in their DevOps practices, not to mention their DevSecOps initiatives.

From her perspective, educating developers as to what is expected of them from a security perspective before a line of code is written is key.

“To help increase compliance, we’ve begun to see organizations taking steps to introduce security tooling into their pipeline process and add testing into the build process,” she said.

Then they’ll add a step that triggers an alert for non-compliant builds but won’t prevent the build. Then they’ll add a step that mandates sign-off by the developer that they’ve seen the security alert for a non-compliant build but still won’t interrupt the build.

“In rare organizations with highly mature DevSecOps processes, they’ll fail the build for non-compliance,” she added.

McLean said organizations need to understand what’s realistic given their maturity and risk tolerance and implement controls that match.

They also need to reevaluate those controls to look for opportunities to increase adherence to security policies and other compliance regulations.

Overall, continuing to educate through compliance automation in the software development life cycle (in development, testing and deployment phases) further helps reinforce best practices while ensuring issues are uncovered before they become a risk or technical debt.

Mackey said because DevOps platforms touch the software powering your business, when choosing any DevOps platform, the security of the platform itself and the competencies it enables should always be must-haves.

“In effect, any decision about new software should be based on how it improves the current security capabilities of the business,” he said.

August 30, 2022
Everything is DevOps

It’s funny how we in IT tend to be absolutists, shoving everything into the current bucket. XML was going to eliminate programming—even though it wasn’t even particularly good at data representation. NoCode was going to eliminate programming (yeah, that’s a theme, I’ll skip the other 50 or so ‘going to end programming’ cycles we’ve been through), AI will eliminate IT, you must have a mobile app, even if no one will ever use it, my OS/browser/language is better than yours and will win—you get the idea.

We’re currently coming to the end of the “Everything is DevOps” phase. DevOps is a great methodology that helped solve several intractable IT problems but, like everything, marketing and a ton of smart people that want to be “right” have taken it too far. Or have they?

The thing with DevOps is that contrary to what those who try to make a separate market out of DevSecOps or SRE try to tell you, it was designed to literally encompass all of human IT activity. We can excuse the DevSecOps crowd, since DevOps’ roots had a definite “Security is an unimportant bottleneck” feel to it. The SRE crowd, on the other hand, appears to be another case of “my silo is special”. Or, better yet, a stompy marketer saying “We’re late to the DevOps game! Let’s invent something just like it, claim we do it more broadly and give it our own name!”

Let’s look at what each does, for those of you who are skeptical. 1) break down silos. Check. 2) Monitor everything. Check. 3) Use monitoring results to improve product performance and stability. Check. 4) Design and implement products and product features. Check. There’s more, but you get the idea.

Do you know what all of this ignores? You.

One of the issues that our absolutism brings about is that marketers get to dictate what is important. Sometimes that is what enterprise IT needs. Far too often it is what the marketer is paid to dictate.

I don’t care if you call it DevOps or SRE, I care that your organization feels like it is getting its money’s worth out of IT—something, I might add, that most organizations did not feel prior to DevOps. At least most of the ones I dealt with. I even had a job where they created a whole new department and hired me to run it to “get around those people in IT.” I spent as much time educating business leaders about not duplicating entire groups like storage services and InfoSec as I did building and deploying new products. So avoiding that kind of issue is near and dear to my heart. And we’re more ‘there’ than we have ever been.

I do care about whether you call it DevSecOps or not. It should not be different. Shifting security left is the appropriate solution to the double-edged sword of security staffing shortages and increased adversary capabilities. So if your developers are getting secure coding feedback in their IDE or each time a build is kicked off, great! But call it DevOps. Say it is part of the DevOps process. Same with configuration reviews like cloud configuration management—it’s just part of the DevOps process, not a security step to be thought of differently.

I could go on, but hopefully I’ve spurred your imagination a bit and don’t need to.

You rock. Streamline your thinking to streamline your processes, ignore absolutism and call it macaroni for all it matters. Just keep doing what needs doing—at the most convenient place to do it. And keep developing, deploying and monitoring the systems that keep the company running.

August 24, 2022