Welcome to The Long View—where we peruse the news of the week and strip it to the essentials. Let’s work out what really matters.
This week: A third-party Instagram app causes ructions, remote work causes office slump, and more on Cali’s salary transparency law. (more…)
Given today’s evolving threat landscape, organizations and businesses in every sector now have a critical need to produce secure software. Criminal gangs, professional attackers and hostile nation-states are employing advanced tactics designed to exploit any vulnerabilities in programs, applications, networks and even raw code. Attackers are constantly finding new ways to bypass even the most advanced protections and defenses. For example, many have shifted their focus from delivering malware to instead compromising credentials, or launching targeted attacks against a supply chain. And while those high-level incursions are happening with much greater frequency, so too are the more basic attacks like ransomware and SQL injections that have been a scourge on cybersecurity defenses for many years. It’s becoming apparent that while cybersecurity platforms and defenses are critical components in defense against modern attacks, what is truly needed is secure code that can be deployed free from vulnerabilities. And that requires security-aware developers with verified security skills.
Most developers say they are willing to champion security and commit to higher standards of code quality and secure output, but they can’t do it without a lot of support, as well as a reworking of the traditional metrics by which they are often judged by their employers and organizations.
Coding best practices have continued to evolve over the years in response to business needs and market trends. In the past, most applications were created using the so-called waterfall development model where software engineers worked to get their code ready to meet an ongoing series of milestones or goals before moving on to the next phase of development. Waterfall tended to support the development of programs that, having met all of the previous milestones along the way, were free from bugs or operational flaws by the time they were ready for the production environment. But it was slow going, with sometimes 18 months or more between starting a project and getting to the finish line.
The Agile method tended to replace Waterfall, putting a much greater emphasis on speed. And this was followed by DevOps, which is built for even more speed by combining development and operations together to ensure that programs are ready for production almost as soon as they clear the final development tweaks.
Putting speed over security—and nearly everything else beyond functionality—was a necessity as the business environment evolved. In a cloud-based world where everyone is online all the time and mobile transactions by the millions can happen every few seconds, getting software deployed and into the continuous integration and continuous delivery (CI/CD) pipeline as quickly as possible is critical for business success.
It’s not that organizations didn’t care about security. It’s just that in the competitive business environment that exists in most industries, speed was seen as more important. And developers who could match that speed thrived to the point where it became the primary means by which their job performance was judged.
Now that advanced attacks are ramping up so dramatically, deploying vulnerable code is becoming a liability. The preference is once again shifting, with security increasingly becoming the primary focus of software development, with speed a close second. Bolting on security after the fact is not only dangerous, it also slows the process of deploying software. That has led to the rise of programs like DevSecOps that attempt to merge speed and security together to help generate secure code. But developers trained for pure speed can’t become security experts without a lot of help and support from their organizations.
The good news is that most developers want to see a shift to secure coding and a reprioritizing of security as part of the development process. In a comprehensive survey conducted by Evans Data of over 1,200 professional developers actively working around the world earlier this year, the overwhelming majority said they were supportive of the concept of creating secure code. Most also expected it to become a priority in their organizations. However, only 8% of the respondents said that writing secure code was easy to accomplish. That leaves a lot of room for improvement within most organizations’ development teams between what is needed, and what is required in order to get there.
Simply mandating secure code won’t get the job done. Development teams need training, support and a change in how software engineers are valued and judged within their organizations.
The biggest thing that is needed is more and better training for them. And it should be customized so that less experienced developers can begin their training by learning how to recognize the kinds of common vulnerabilities that often creep into code, with lots of hands-on learning and examples. Meanwhile, more advanced developers who demonstrate their security skills can instead be tasked with things like advanced threat modeling concepts.
Teamwork should also be emphasized so that the developer community can help one another grow their skills. Skilled and willing developers who know security should be appointed as security champions. Their responsibility as a champion will be to help fellow developers improve their skills. And while a security champion is almost always an informal title, they should be given the respect, rewards, and compensation that such an important position deserves.
In addition to funding and supporting training programs and security champions, including giving developers enough time away from coding in order to properly participate in those programs, organizations also need to change the way that developers are evaluated. The primary metric for rewarding developers needs to shift away from raw speed. Instead, evaluations could reward those developers who can create secure code that is free from vulnerabilities or exploits. Yes, speed can be an evaluated factor as well, but first and foremost, code needs to be secure and free from vulnerabilities.
Producing insecure or vulnerable code is no longer an acceptable risk for most businesses. And bolting on security after the fact is becoming increasingly ineffective. Thankfully, the best weapon to fight this disturbing trend is having the developer community produce secure code that attackers can’t exploit. Most developers are willing to step up to that challenge. They just need the proper training and support to help make it happen.
For nearly the entire history of application development—certainly since application development became available to a larger market with the adoption of the personal computer for expansive business use—testing has taken a back seat across most of the industry. This is not because we don’t care about code quality but because it takes resources away from developing cool new tools—or even products—and slows down release cycles. Historically, both of these issues have been huge. Manual testing staff grows quickly from a few people to a massive department and waiting for that process to complete could take forever.
So, we’ve not done a lot of it. Oh, when executives or someone in leadership demanded more thorough testing, IT management would promise to do so and then—to fit within budget and time constraints—would tell developers to test their own code. There are several areas where you don’t ask the person responsible for creating the final product to review that final product—because if they knew they were doing something wrong, they’d fix it. Writing is one such area—that’s why good editors are worth their weight in gold, even in the age of the internet where anyone can (and too many do) spew onto the digital page whatever they like without the benefit of an editor. Another is testing application code. I test as I write my code, and like every person who has spent time as a professional developer, I have made a couple of stupendous errors. If I’d known that the particular weakness I left in the system was there, I would not have said it was done.
When DevOps came along, I expected things to change—until I understood that DevOps (and Agile, in lockstep) was all about delivering faster, not better. Anything that slowed down the process—testing, security, even compliance, to an extent, that cannot be avoided—was frowned upon. When test-driven development (TDD) came along, I instantly developed a love/hate relationship with it. We used it for a project I was on and, frankly, it shifted quality concerns to the tests—the code could be forced to pass all of the tests, but how thorough and accurate the tests themselves were—particularly as the project grew and changed—became the question. Done right, TDD would solve a lot of our testing problems. Heck, even done wrong it would have taken care of most problems that simple unit testing could cover.
But after decades, I think we are finally at the point where we can demand and require thorough testing with acceptable and growing coverage numbers. And you need to consider it. Test automation has come a long way and far too many of you haven’t come along with it. If you struggle to justify investing in test suites, I get it. I’ve been the manager that had to choose between improving DB performance and testing. But the modern iteration includes security testing. So if, out-of-the-box, you could get a tool that checked for valid inputs and alerted to places where invalid data might break the application, that might help. And that’s just the simplest case. I’m currently working in an adjacent space, so I don’t want to get too crazy selling you here. I’ll just say, “Testing isn’t what it was even a couple of years ago. Check it out.” Both application and security testing. And the changes will continue to come as claims of using AI/ML turn into applying actual algorithms that your AI or data science people can compare to help choose a high-quality product.
We could debate all day whether DevOps made apps more or less stable and secure, but we don’t need to. Finally, continuous testing is real. Check out the vendors doing this, compare them and see how easy it is to fit them into your environment. More stable, more secure, not the hold-up you have stuck in your head. What’s not to like? You are already rocking it; automated testing could make you look even better.
Welcome to The Long View—where we peruse the news of the week and strip it to the essentials. Let’s work out what really matters.
This week: Wipro fires 300 for moonlighting at competitors, Python has a nasty 15-year-old bug, and companies are finding new ways to lay people off without calling it a “layoff.” (more…)
I’ve made no secret of the fact that DevOps was a game-changing advance in how the business of IT was done. But people tend to get religious about the methodology and forget the point. That’s okay, DevOps has its roots in that mentality. After years of thinking bad things about the authors of The Phoenix Project for portraying anything that slows the process–you know, like code quality or security–as bad, I have come to accept that absolutes were required to drive forward motion in the market. “We’ve always done it this way!” was king in IT, but the rest of the business was fed up. That book got the ball rolling and we’ve slowly but surely pushed back against the mentality that adding milliseconds (or even a week) to build times in the average enterprise has any impact worth mentioning. Oh, some people still claim it does, but we can convincingly make the code quality/security argument.
It is funny to me that the first few years of DevOps were focused on “We’re gonna change your religion!”—wanting to force upheaval of the entire IT org chart without a thought to the fact that you still needed specialists in storage, ops, etc. to keep the lights on. The mentality was that development issues were a people problem. To a large extent, they were right; they just missed one people problem in the set of equations. Let’s look at the real pipeline that we all don’t talk about, but really should:
Notice there aren’t tools in this chain; instead, there are groups of people. It is sad but true that the last two–representing acceptance testing and customer rollout–are almost always merged these days in all but a few organizations.
This isn’t controversial; we still do the dev bit largely before the ops bit; the business has to decide what we want to offer the customer and customers are at the end of the line. And that is the key. Customers are at the end of the line. Now, I get to make you uncomfortable. Nothing in the above diagram matters except the first and last arrows. Think of it like this:
What’s important in what we do comes down to two questions: “What does the business unit want to offer customers?” and “What do the customers expect of the business unit?” In this sense, “business unit” might be IT and “customers” might be co-workers. That changes nothing. What the business unit offers and what customers want is all that matters. The rest is implementation detail.
Don’t believe me? It is an easy case to prove. Let’s try this one, with or without IT involvement, depending upon the problem being solved and the organization:
The business unit determines what they want to offer, customers let their demands be known (one way or another) and the business offers the solution they think fits for the customers. As I said above, the rest is implementation details.
What is your point?
This leads me to the inevitable observation. Testing and security are:
So, your toolchains need to include stability and security tools—I would argue at every step of the process, but where and how much is really a business call. My security peeps will get upset that I called security a business call, but it is. You can choose to be non-compliant and insecure; the impact of those decisions on your organization is all on you. OTOH, of all the things that the organization (both the business generally and IT specifically) are focused on, you can afford to drop a few and get security and stability right the first time.
Meanwhile, you’re keeping things running while we debate things like this. While I repeatedly refer to a large portion of my target audience as “implementation detail,” I’m sure you all know how much I (and many others) appreciate the rockstar job you’re doing keeping the IT org supporting the greater org.
An open source GitBOM tool, discussed at the Open Source Summit Europe conference this week, can automatically track every source code file incorporated into each built artifact.
Nell Shamrell-Harrington, a principal software engineer for Microsoft, told conference attendees via a video link that the GitBOM tool, based on a compact Artifact Dependency Graph (ADG) technology, can verify which artifacts are being used across any programming languages, environments and packaging formats without any effort made by developers. To do this, GitBOM repurposes the directed acyclic graph made available in Git version control software.
All that is required is for GitBOM identifier—a unique, content-addressable reference—to be inserted into the artifact at build time. That identifier then creates a Gitoid that can be consumed by ADG, Shamrell-Harrington said. Each Gitoid then views an artifact as a binary large object (BLOB) through which of all the components that went into it, including source files, dependencies and object files can be identified.
Support is also included for the Software Package Data Exchange 2.3 format for sharing software bill of material (SBOM) information discovered by GitBOM.
GitBOM can also detect potential vulnerabilities at runtime, regardless of the depth, in every software artifact that might be impacted to make it simpler for IT teams to remediate issues. That approach makes it simpler to find, for example, every instance of the Log4j vulnerability that might exist in an application environment. GitBOM only includes the minimum information required to create a fingerprint to enable efficient runtime scanning for a known-vulnerable artifact.
The goal is not to replace SBOM tools but rather to make it easier for them to collect data about artifacts in the most frictionless way possible for developers, Shamrell-Harrington said. GitBOM enables any metadata to be linked to a specific set of corresponding software artifacts to make it easier to discover exactly how they were constructed. Artifact IDs can then be extended to produce GitBOM documents that can be consumed by other build tools.
GitBOM is being shared at a time when the focus on securing software supply chains has never been higher. In the wake of the Log4j vulnerability disclosure, the Biden administration is requiring every federal agency to construct SBOMs to enable developers to find instances of vulnerable software components. Most large enterprise organizations are generally following suit.
However, collecting SBOMs is only the first part of a bigger challenge. Organizations will then need to acquire tools to analyze SBOMs and provide feedback to application developers. In effect, eventually, organizations will be able to accept or reject software based on the components being used to create it. Armed with those insights, developers then have the option to upgrade software components to bring applications in line with any policy an organization defines.
It may be a while before SBOMs proliferate across IT organizations, but the immediate issue at hand is simply finding the least-intrusive way possible to create them.
Welcome to The Long View—where we peruse the news of the week and strip it to the essentials. Let’s work out what really matters.
This week: Twitter is in a “non-redundant state” thanks to a hot summer, AI is likely to eliminate us, and Patreon fires 80 staff amid nasty allegations (TW: Child safety). (more…)
A survey of 16,510 IT and IT security practitioners published today by Rezilion, a provider of a platform for automating the remediation of software vulnerabilities, found nearly half of respondents (47%) worked for organizations that have a backlog of vulnerable applications. More than half (66%) said their backlog consisted of more than 100,000 vulnerabilities and the average number of vulnerabilities that need to be remediated is 1.1 million.
The survey, conducted by the Ponemon Institute, also found more than half of respondents ( 54%) said they can only patch less than 50% of the vulnerabilities.
Factors that keep teams from remediating included an inability to prioritize what needs to be fixed (47%), a lack of effective tools (43%), a lack of resources (38%) and not enough information about risks that would exploit vulnerabilities (45%).
More than a quarter (28%) also said remediation is too time-consuming. The survey found that 77% of respondents said it takes longer than 21 minutes to detect, prioritize and remediate just one vulnerability in production.
In addition, 85% said they spend more than 16 minutes detecting one vulnerability in development. A full 82% also noted it takes longer than 21 minutes to remediate one vulnerability in development. A majority of respondents said it is either very difficult (36%) or difficult (25%) to remediate vulnerabilities in applications.
On the plus side, more than half (56%) said they use automation for vulnerability remediation, with 43% reporting times to respond are significantly shorter as a result.
Rezilion CEO Liran Tancman said the survey makes it clear there’s still plenty of room for improvement. The number of vulnerabilities that need to be addressed is staggering, he said. IT teams need to focus their immediate efforts on identifying the vulnerabilities that are the most critical, he noted.
Remediating those vulnerabilities will soon become a higher priority as more regulations focused on securing software supply chains come into effect. In fact, the software bills of material (SBOMs) that these regulations require will expose much of the dirty laundry that currently exists in application environments, said Tancman.
Ultimately, the goal needs to be enabling organizations to address vulnerability issues without slowing down the rate at which applications are being built and deployed, Tancman added. The degree to which that goal can be achieved will vary widely as more responsibility for security is shifted left toward application developers. However, the need to provide tools to developers that enable them to address vulnerability issues has never been more apparent, noted Tancman.
It’s clear the level of application security vulnerability debt that organizations will need to address can no longer be ignored. The issue now is to determine how best to resolve these issues either by fixing existing applications or replacing them with new ones. Either way, it’s clear that there is a need to adopt DevSecOps best practices to better secure software.
In the meantime, as application security becomes a boardroom-level issue, there be a lot more scrutiny applied all across the software supply chain.
Today, IT pros field an increasing number of assets they need to monitor to secure systems and understand what’s happening in their database and applications. This is further complicated as organizations adopt hybrid IT strategies and IT pros are tasked with managing tech stacks that span across on-premises and public clouds.
To cut through complexity and create end-to-end visibility into their IT systems, organizations must enact an observability approach.
When it comes to observability, many think it means the same as monitoring. Whereas monitoring is a verb; observability is a measure of how well the internal states and conditions of a system can be inferred from its external outputs.
When monitoring systems are set up, dashboards are built around certain assumptions that an IT pro may expect will happen. When monitoring databases, teams are building multi-level component IP systems. The algorithms must process that data, and the user interface must interact with it.
However, to make that system highly observable, performance information and other kinds of metrics across all kinds of different endpoints must be incorporated. Especially since IT pros must deal with microservices, mesh, IoT or containers orchestrated through Kubernetes.
In the past, IT pros would typically monitor a SQL Server or even other database platforms like PostgreSQL or MySQL. In this new era of observability, on the other hand, IT pros need to do something different with the front-end code with regard to the different kinds of algorithms processed against that data set. Typically, developer teams would write the code, but there wasn’t anything written in that code to send out debug signals.
Today that has changed. To make sure our systems are fully observable (because we can’t anticipate all possible problems!), we must log different steps to keep track of the many moving pieces. Organizations must collect as much of everything as possible. The more data, the more information organizations have about what’s happening in their application, from the database through the virtualization or containerization layer to the virtualization host or the orchestration to the bare metal. If organizations can get visibility into those elements, we can enable our systems to be fully observable and more easily diagnose any problem or anomaly that happens.
When using the observability approach, high-level dashboards are handy. They can help IT pros roll up everything that’s happening and get to a root cause analysis if there is a problem. However, sometimes there’s not a specific problem; there’s just a general performance issue. With observability platforms, teams can quickly discover, map and understand dependencies in ways that make it easier to achieve and maintain service level objectives (SLOs).
As IT teams increasingly distinguish between merely monitoring IT environments and achieving observability, they can proactively surface anomalies using machine learning algorithms and other forms of data science. And the depth and insight that can be applied to managing IT is increasing dramatically. Observability is quickly becoming table stakes for any modern IT organization.
Ever since the SolarWinds attack back in December 2020, software supply chain attacks have been top-of-mind for any company that builds software. The idea of endangering not just your organization by being attacked but also your customers really sharpens your focus.
Another complicating factor is that applications use a combination of open software components and homegrown code to deliver applications. This software supply chain means your software can be compromised through no fault of your own but instead based on the components you use. The Log4j attack is an example of a component creating vulnerabilities in any software that uses it. So how do we defend our software? We need to think differently about where security starts and how security concepts can be integrated into software earlier in the development process where it interfaces with the rest of the technology infrastructure.
Modern software development is built on the CI/CD pipeline. It allows the application security testing and software composition analysis to happen way earlier in the integration process (shift left), and that allows you to identify and fix issues before the code gets anywhere near customers. Sounds simple, no? Just run your code through a pipeline, integrate security testing, identify issues, fix them, and deploy secure software.
As with most things, what sounds simple in theory can be challenging to implement in reality. First, shifting left requires a collaboration between developers and security teams. Finding the security issues don’t matter if they don’t get fixed. To increase the likelihood of these issues getting fixed, the security team needs to provide context to the developers regarding the urgency of the flaw or vulnerability and some guidance on the best way to fix the issue. To be successful, security folks can’t just drop a report with hundreds of defects and vulnerabilities on the developers and expect that anything productive is going to happen. If this scenario reminds you of getting a report from your vulnerability scanner with hundreds of issues you won’t fix, well, that’s because it’s very similar.
Second, security must add value to the development and architecture process by offering up infrastructure-as-code (IaC) templates and a toolchain that makes it easy to implement secure application stacks. Maybe that looks like a template library managed by a cloud security Center of Excellence, which can provide these tools across all of the application teams leveraging efforts.
Finally, you can implement a security champions program, training a developer (or two) on each team about security issues and they can act as a security emissary to the developers. This provides scale, ensuring that the security team doesn’t become a bottleneck to fixing the security issues.
Given the sophistication of the software, the attack surface of the code, and the velocity of the development process, combatting supply chain attacks requires tight and ongoing collaboration with the development team and shifting left to catch the security issues earlier in the integration process.
Want to learn more about this and other related topics? Join CloudBees for DevOps World 2022, held in Orlando, Florida, at the World Marriott Center. Use discount code DW22 when registering and get access to all the fantastic offerings, including keynotes, sessions, training and other interactive activities. We can’t wait to see you there for the DevOps Remix!