devopsy.kinsta.cloud

Tag: software updates

Why Nano Updates Only Work if You Begin with the Latest and Greatest Software
Is there a silent threat of technical debt looming in your organization? You probably should take a deeper look.

Modern software systems are composed of hundreds of interdependent components. How are you updating those components – system libraries, base containers, and app packages? Do it poorly, and you will trigger unintended side effects, like failing CI pipelines, hours of debugging, or even outages.

Ironically, these risks cause teams to delay updates, even when they fix critical vulnerabilities. According to recent research, 60% of known vulnerabilities remain unaddressed in the average organization today.

The solution to reducing technical debt is simple: consistent nano updates – precise, minimal changes that reduce risk while maintaining agility. However, this strategy works best when you’re starting with the latest and greatest software from the beginning. Let’s explore how implementing continuous nano updates and starting with the latest software can reduce technical debt before it starts to accumulate.

What are Continuous Nano Updates – and Why are They Different?

Nano updates are surgical upgrades that continuously apply the smallest possible change needed to update systems. This eliminates security vulnerabilities and reduces the risk of unintended side effects while improving control and traceability. Developers replace a single vulnerable dependency in a container without altering the rest of the image, or update a package version while preserving all surrounding libraries and configurations.

Nano updates allow for continuous delivery, improved collaboration and faster triage and recovery when something does break. Knowing these distinctions can make or break your approach to updates and ensure they roll out successfully.

Starting with the Latest is a Prerequisite

Many organizations assume they’re “up to date” simply because they use mainstream Linux distributions like Red Hat or Ubuntu. In reality, these traditional Linux distributions ship a snapshot in time, typically from two to five years ago. These are outdated base images, with lagging dependencies and contribute to accumulating technical debt throughout their entire organizations. Using a one-year-old image doesn’t qualify as the latest version because it’s often hundreds of upstream releases behind, leaving hundreds of CVEs unpatched.

Nano updates only truly work when you’re already on the current software stacks. Otherwise, small updates are impractical and will have outsized effects due to compounding code drift from legacy package versions. For example, updating a base container image might pull in multiple outdated packages with incompatible and conflicting versions, yielding issues and breaking integrations. If you haven’t modernized your foundation, you can’t build a nano update strategy on top of it. It’s critical to think about this early on and ensure the foundation is there.

Ending the Tug-of-War Between Security and Engineering Teams

Once the latest and greatest versions of all tools are in place, engineering teams often operate in a steady state of “maintaining and monitoring.” At the same time, security teams are focused on making upgrades – and understanding the risks they pose – to protect the organization from further risks. By making regular nano updates, together engineering and security teams, can prevent longer-term risks and issues from the start.

Some healthcare organizations, for example, may have long periods of software freezes like during open enrollment. Broad updates are risky during this time, but nano updates enable teams to patch vulnerabilities and avoid the most destabilizing environments. The result is that patient-critical systems remain both secure and operational. Taking this approach helps bridge the gap between security and engineering teams and ensures that technical debt isn’t looming throughout the organization.

Helping Developers get a Leg Up by Starting Clean and Staying Clean

Best-in-class developers are adopting nano updates as part of modern CI/CD workflows and seeing benefits from continuous delivery as smaller incremental changes facilitate more frequent, reliable deployments. They’re also improving collaboration across their development and operations teams and seeing faster recovery as pinpointing and reverting specific changes to issues becomes more straightforward.

However, developers in legacy organizations often struggle to adopt this mindset. To bridge that gap, there are four steps they can take to set themselves up for success:
1. Rebase first: Start by modernizing base containers and dependencies. Without this, truly nano updates aren’t really possible.
2. Automate dependency management: Implement monitoring to detect and assess the impact of changes in real-time.
3. Monitor changes closely: Track the effect of each nano update using observability and alerting, rolling back quickly and triaging regressions easily.
4. Educate the team: Foster a culture that values incrementalism and sees continuous nano updates as a way to move faster with less risk.
Engineering teams can’t afford to delay updates, but they also can’t risk instability. Nano updates give them the best of both worlds – but only after they’ve adopted the latest software versions as a baseline.

The real challenge isn’t just applying updates – it’s building a culture of discipline around starting clean and maintaining great hygiene. With this approach, organizations avoid unnecessary spending, reduce CVE exposure, empower developers with the latest software features, and move confidently in a complex, fast-moving world free of technical debt.
October 23, 2025
The Day a Software Update Broke Corporate America
This July, I watched the corporate world almost screech to a halt. All because of a tiny software update. One small change to CrowdStrike’s security software turned 8.5 million Windows computers into very expensive paperweights.

Let’s start at the beginning. It was a regular Friday morning — people were getting their coffees, checking their emails, going about their usual nine-to-five shuffles. When suddenly, everything went sideways. Banks couldn’t process transactions. Airports turned into chaos (more than usual, if you can believe it). Even hospitals had to postpone surgeries.

The price tag for this mess? $5.4 billion just to the Fortune 500 alone; in one day. All because of a small software bug. Let that sink in for a minute.

The Hard Truth About Software

Here’s the thing — as someone who has spent years in tech, I can tell you that software bugs are inevitable. Sure, we have amazing tools now — AI, ML and automated testing. But perfect software, from day one and every time an update is made? No, not happening.

But maybe we are looking at this wrong. Instead of chasing the perfect software, what if we set up our systems more smartly?

Since 2017, Microsoft SQL Server has been able to run on Linux. That is a big deal. But most companies haven’t taken advantage of this. They are still running everything on Windows, putting all their eggs in one basket (platform), so to speak.

What Your Company Needs

Think about it like this — you wouldn’t keep your retirement savings in just one investment. The same principle applies here. What you really need is:
- The ability to seamlessly manage workloads across both Windows and Linux
- HA automation to quickly switch between platforms when things go wrong
- Rock-solid security that works everywhere
With automated, cross-platform failover capabilities, critical systems can continue to operate no matter what underlying OS issues arise. Instead of panicking, your business would run as usual.

After watching the CrowdStrike situation unfold (and helping clean up the mess), I can say one thing with absolute certainty — relying too heavily on a single platform is a risk none of us can afford anymore. The technology to build more resilient systems exists today.

The Solution is Here

For organizations prioritizing resilience, adopting infrastructure-agnostic solutions that enable seamless workload management across platforms should be at the top of the list. These technologies empower businesses to stay operational, secure and confident no matter the environment. We just need to use them.
December 6, 2024
DevOps Must Learn From CrowdStrike’s Outage
The CrowdStrike outage on July 19, 2024, is a stark reminder of DevOps practices’ critical role in deploying updates to maintain the security and reliability of applications and systems. While the underlying software defect was the immediate cause, the broader issue lies in the deployment process that allowed a severe flaw to impact a global customer base.

Key Takeaway: To maintain trust and reliability in today’s complex software, security, cloud and data center ecosystem, we must prioritize robust, measured deployment strategies.

The ability to rapidly deploy software updates across large, diverse environments is necessary for many software offerings. Updates can have unintended and, rarely, catastrophic consequences if done improperly or without diligent governance. The CrowdStrike incident highlights the importance of adopting more sophisticated deployment strategies—such as staggered, A-B, canary and phased rollouts—to minimize the blast radius of any potential defects. By initially releasing updates to a smaller subset of systems, organizations can validate changes in real-world conditions, catching issues before they affect the entire user base. Then, the update can be released in stages to increase the number of customers. This approach significantly reduces the risk of widespread disruptions and ensures that problems are contained and addressed quickly.

Additional Resources
- Research Note: The CrowdStrike Outage – A Detailed Post-Mortem
- Research Note: CrowdStrike IT Outage: Critical Global Impact and Implications for Cybersecurity
Let’s move beyond this single CrowdStrike incident and focus on the lessons we need to learn to avoid repeating them in the future. As software delivery becomes faster and more automated, the need for meticulous, well-orchestrated deployment processes has never been greater. It’s not just about preventing the next outage; it’s about building a culture of reliability and accountability in every step and layer of the software supply chain. The urgency of this need cannot be overstated.

Customers of software vendors bear a crucial responsibility in ensuring their systems remain secure and reliable. They should not rely on automatic updates without oversight, as doing so can leave their environments vulnerable to unforeseen issues. Organizations need to implement their own vetting and validation processes, testing updates in controlled environments before widespread deployment. This added layer of scrutiny helps catch potential problems early and prevents disruptions from unquestioningly trusting every update. In this way, customers play an active role in maintaining the integrity of their systems and minimizing risks.

This is our call to action: To maintain trust and reliability in today’s complex software, security, cloud and data center ecosystem, we must prioritize robust, measured deployment strategies.

Mitch Ashley is Chief Technology Advisor with The Futurum Group and CTO of Techstrong Group’s tech media platforms covering DevOps, cybersecurity, AI, cloud native, cloud infrastructure, platforms and ITSM.

Mitch’s analyst research is available on FuturumGroup.com and TechstrongResearch.com.
September 1, 2024
CNCF Graduates TUF Project to Secure Software Updates

The Cloud Native Computing Foundation (CNCF) announced today that an open source specification for securing software update systems has graduated to becoming a top-level project.

The Update Framework (TUF) is made up of a set of libraries, file formats and utilities that can authenticate files and images before they are downloaded from a software repository. Justin Cappos, associate professor of computer science and engineering at NYU Tandon School of Engineering who led the development of TUF, said the framework enables IT organizations to better secure their software supply chains.

In addition, TUF isolates artifacts in a repository in a modular fashion that not only limits the amount of damage that can be inflicted by a breach of the repository, but Cappos said it also makes it easier for IT organizations to roll that repository back to its last known good state.

For a CNCF project to graduate it needs to demonstrate thriving adoption, an open governance process and a strong commitment to community, sustainability and inclusivity. TUF was developed at the University of Washington in 2009 by Cappos and Justin Samuel. TUF has been based at New York University Tandon School of Engineering, which makes a reference implementation available to any interested party, since 2011. It became a CNCF project in 2017 and is now the first security-oriented project to graduate among the eight other CNCF projects that have graduated.

TUF is already employed within Notary, a separate CNCF project that provides a mechanism to both certify the validity of the sources of Docker images and encrypt the content of those images. TUF is also employed within Uptane, which is employed to deliver software updates to automobiles. Other organizations using TUF in production environments include Amazon, VMware, Google, IBM, Red Hat, Datadog and DigitalOcean.

Cappos said he would like to see TUF adopted more broadly to secure software supply chains in medical systems as well as critical infrastructure systems such as the power grid. As cyberattacks targeting these systems increase, Cappos said it’s apparent there is a need for a more resilient approach to securing software running on these types of platforms.

Longer-term, as organizations embrace best DevSecOps processes, it’s now only a matter of time before organizations focus more of their time and energy on securing the software supply chain. However, Cappos said right now there is too much focus on trying to discover application dependencies and scanning images. While those are important tactical tools, Cappos said there needs to be more focus on strategic approaches to securing software supply chains.

It’s not clear to what degree DevSecOps will ultimately push organizations to embrace a more holistic approach to building and managing software supply chains or whether larger concerns about the security of software will push more organizations to embrace DevSecOps. Regardless of whether the impetus for building and deploying more secure software comes from the top down or the bottom up within an organization, the important thing is that in terms of tools and processes rapid advances are finally being made.

— Mike Vizard

December 18, 2019
How Often Should You Release with Continuous Delivery?
If you do continuous delivery, you could release software updates as frequently as you want. But should you? Here’s what to consider when deciding how often to release.

Although the terms continuous delivery and continuous deployment are often used interchangeably, they don’t mean the same thing. Under continuous delivery, you prepare all software updates as potential candidates for release. But continuous delivery on its own does not mean that you release all of those updates into production as soon as they are complete. You would have to adopt continuous deployment to do that.

So, if you’re doing continuous delivery, you have the option of releasing as frequently as you like—every day, every month or even every hour. Determining how often you actually release requires weighing several important factors, including:
- How important are your updates? If changes to your app mostly involve mundane things such as interface tweaks, continuous deployment probably will confuse users more than please them. They will have to adapt constantly to new features. But if you can roll out big, user-friendly changes on a continual basis, then deploying continuously can help keep your users happy.
- Is there a marketing advantage to be gained by waiting longer to release? Sometimes, advertising new features but waiting a while to release them can help build excitement. Case in point: Apple Inc.
- Do other apps depend on your app? If you’re working within an ecosystem where your app integrates with other apps or platforms, continuous deployment makes the lives of partner developers more difficult because it requires them to adjust constantly to changes in your app. To help them ensure their work will always be compatible with yours, provide space between your releases.
- Is your development pace consistent? If not, continuous deployment can be problematic because users will see rapid updates at some times, but stagnation at other times. Continuous deployment only works well for users if they can count on a relatively steady stream of updates.
- Do your updates create a learning curve? If users will have to learn new tricks to use an updated app, continuous deployment may create more confusion than happiness.
- Can multiple versions of your app coexist? If all users need to be running the same version of your app in order to be current, continuous deployment can help assure that they are. But if you can support multiple versions of the same app at once, less frequent releases should work for you.
If you can’t decide how frequently to release, you might consider releasing on a fixed time schedule—once every two months, every six months, every year or whatever works for you. This is what Canonical has done with Ubuntu Linux for a long time: It releases a new version of the operating system on the second Thursday of October and the fourth Thursday of April each year (with a few exceptions). This system assures that developers and users always know the schedule for a new release.

Whatever approach you take, the most important thing is to realize that continuous delivery does not require you to release as often as you can. It gives you the flexibility to release as often as you want. Deciding how often that should be is up to you.

— Chris Tozzi
December 16, 2016