devopsy.kinsta.cloud

Category: DevSecOps

DevSecOps Will Cross the Chasm in 2022

We’ve been talking about DevSecOps and shift-left security for years. Although this approach probably didn’t “cross the chasm” in 2021, we did see some very telling milestones. Cybersecurity VC funding surged to record heights with a focus on DevOps and cloud security and the need for DevSecOps became glaringly evident with the Log4j vulnerability. On a more personal note, our own open source security project Checkov has surpassed three million downloads.

Much of this momentum was inevitable. Those in infrastructure and reliability circles have already been adopting and touting the benefits of shifting left and automating as much as possible, so in many ways, it’s expected that security would follow suit. Undoubtedly, the pandemic has also accelerated the need for developers to work more autonomously without having security personnel and processes acting as barriers to productivity and velocity.

We’re confident that 2022 will see the silos between development and security teams continue to crumble as developer-led security practices become the norm for cloud-native organizations.

DevSecOps finally crossing the chasm doesn’t mean that every enterprise and traditional organization will shift security left and adopt security best practices. It will, however, give them competitive advantages over those who don’t—both in decreased security costs and increased developer productivity and, thus, time-to-market.

What else is in store for the future of DevSecOps?

Rise of the DevSecOps Job Title

Fewer than 5,000 people on LinkedIn currently have “DevSecOps” in their job title, yet there are over 20,000 current openings for DevSecOps roles. In 2022, expect to see more of those positions filled. This means security teams across industries will conduct fewer manual security audits, there will be a considerable influx of homegrown DevSecOps tooling and point solutions will likely begin to consolidate into single platforms.

Blurred Lines Between Application and Infrastructure Security

Until recently, application security was a very well-defined (albeit fractured) space focusing on securing the custom code and open source packages that make up applications. However, with the rapid adoption of cloud-native applications, the lines between application and infrastructure security are blurring. We expect to see this trend continue as more engineers take on more infrastructure-related projects, vendors start catering to use cases outside of their core competencies (through acquisitions and in-house development) and the role of DevSecOps continues to expand within organizations.

Infrastructure-as-Code: The Great Cloud Migration’s Next Chapter

We’ve been talking about the great migration to the cloud for years. At this point, many companies are ready to move on to the next chapter: Infrastructure-as-code (IaC). As a result of this trend, DevSecOps will become much more important, as security needs to be baked earlier in development phases or risk being left behind. In addition, security teams will need to become more well-versed in development technologies and practices to provide the proper guidance for the new way applications are built and deployed.

More Software Supply Chain Attacks

Hackers have been targeting retailers and security vendors for years, exploiting a minor weakness to gain access and move laterally into sensitive data. In the past year, however, software supply chain attacks stepped into the spotlight due to multiple supply chain attacks. We expect this trend to continue, which will, in turn, put more focus on securing supply chains. DevSecOps, which has focused mostly on tools and practices for securing the code and infrastructure, will expand to include the supply chain mechanism.

If 2021 was the year of hype for DevSecOps, we believe (and hope) that these best practices are embraced en masse across industries. The benefits of shift-left security are well documented: The number of high severity incidents is significantly reduced, the potential attack surface is minimized, compliance efforts are simplified and the time to remediation is lowered. Organizations also save money by catching misconfigurations and vulnerabilities earlier in the software development life cycle while at the same time gaining time back with tools, both open source and commercial, that are empowering developers to move fast and build applications that are more secure and reliable.

January 5, 2022
Best of 2021 – How to Revoke JSON Web Tokens (JWTs)
As we close out 2021, we at devopsy.kinsta.cloud wanted to highlight the most popular articles of the year. Following is the seventeenth in our series of the Best of 2021.

One of the most common questions about JSON Web Tokens (JWTs): Once they’re issued, how can they be revoked?

What Are JWTs?

JSON Web Tokens are portable, industry-standard identity tokens. They are issued after a login request by a central identity server and used to identify and credential a user and grant access to resources. They can be presented by clients such as browsers and external programs. Applications and APIs can examine them to ensure the caller is properly authenticated and authorized.

The Challenges Involved

One of the most common questions is how to revoke them. The problem is that once they are issued, no further communication is needed with the identity server. That’s one major benefit of JWTs decentralization.

Let’s say you use RSA public/private key signing for secure data transmission. After the IdP signs a JWT using the private key, any service that has the public key can verify the integrity of the token.

Let’s use the Todo-Backend API as an example. The architecture might look something like this:

Now, Todo-Backend can use JWT and the public key to verify the user’s ID and other attributes, and then use that ID to perform operations on that user’s data. Once the JWT has been issued, Todo-Backend doesn’t communicate with the identity provider at all. As a result, if an administrator has locked or deleted that user’s account in the identity provider, Todo-Backend won’t know about it.

Managing Revocations Using a Distributed Event System

The most common way to revoke access to resources protected by a JWT involves setting its duration to a short period of time and revoking the refresh token so that the user can’t generate a new token. This does not revoke the JWT per se; it does solve the root issue, which is to limit access.

In other words, you can set the JWT’s expiration duration to a short period (e.g., anywhere from a few seconds to, say, ten minutes) and set the refresh token’s expiration duration to a longer period (e.g., a two-week or two-month window).

During the normal flow, the Todo API would accept the JWT until it expires, at which time access would be denied. At that time the client could present the refresh token to the identity provider, which would send a new JWT to the client. That new JWT would be valid for another ten minutes.

An administrator can, at the identity provider, revoke the refresh token at any time. Any subsequent request for a new JWT by a client holding that refresh token would fail and access to the Todo API would be lost.

However, while ten minutes isn’t long, this timeframe still comes with risks.

One strategic way to handle this challenge is to set up a distributed event system that notifies services when refresh tokens have been revoked. Upon receiving this event broadcast, backends/services would update a local cache that lists users with revoked refresh tokens. After a JWT is verified, this cache would then be checked to determine whether or not access should be revoked.

This distributed event method truly revokes the JWT, rather than simply waiting for it to expire, as mentioned above.

An Example of Revoking JWTs

Here’s a sample solution that provides support for events and webhooks that can be used to implement this revocation strategy. In this example, when a user logs out or has their refresh token otherwise revoked, a jwt.refresh-token.revoke event is emitted. This includes the user ID and the application for which the revocation occurred. It also includes the JWT lifespan for this application (in the below case, 600 seconds). Any JWT for this user that was issued before 600 seconds from the point in time when this event was received is thus revoked.

For example:
- User logs in, and JWT ‘abc’ is issued at 1:11pm. JWT is good until 1:21
- User logs out at 1:15. Entry is added stating that any JWT for this application and user expiring before 1:25 is revoked.
- Application tries to use JWT ‘abc’ at 1:16. Todo-Backend does not deliver data because this JWT is on the denylist.
- User logs in, and a new JWT ‘def’ is issued at 1:18. JWT is good until 1:28.
- Application tries to use JWT ‘def’ at 1:19. Todo-Backend delivers data because this JWT is good until 1:28 and is therefore not on the denylist.
What you need to do is write a simple webhook that will receive this event and inform JWTManager that the refresh token for applicationId for this user has been revoked.

JWTManager maintains a list of user IDs whose JWTs are revoked. You could also use JWT IDs (the `jti` claim), which, in some cases, might be simpler. JWTManager also launches a thread to clean up the cache to remove expired users and avoid running out of memory.

For each API call, in addition to all the other checks that should be performed against a JWT (verifying the signature, expiration, issuer and so on), the backend needs to check validity with JWTManager.

As a final step, configure the webhook:

The Revocation Process, Simplified

Using the process described above, you can revoke a user’s refresh token and broadcast the event using a webhook. The webhook receivers then update JWTManager, which will render JWTs for that user unusable.

The good news is that you can use a webhook and event system to build this feature into your application quickly. You can also write JWTManager implementations into Java, Node and other client libraries for future use. Other languages, such as Ruby, have denylist implementations already written.

Ultimately, all that’s required are the use of refresh tokens, an API that enables revocation of refresh tokens and a system for distributing the revocation event. This solution should work well with most systems, even large systems with numerous backends.
December 28, 2021
Best of 2021 – From Agile to DevOps to DevSecOps: The Next Evolution
As we close out 2021, we at devopsy.kinsta.cloud wanted to highlight the most popular articles of the year. Following is the ninth in our series of the Best of 2021.

As the workforce shifts to remote locations, networks are becoming more diffuse as the edge gets farther away. At the same time, the number of devices and applications on those networks continues to rise. Some estimates suggest that, by 2030, there will be 15 connected devices for every person on the planet.

The rapid expansion of the remote workforce and their associated application needs has created a significant challenge for both application developers and the IT professionals tasked with deployment and maintenance of applications. Finding the right balance between speed of release and stability and reliability of applications – never an easy task – is paramount for the success of any organization.

Agile and DevOps evolved to help organizations strike this balance, providing the best possible customer experience in the shortest realistic time frame. But have Agile and DevOps fulfilled that promise?

The Promise of the Agile Manifesto

While Agile as a methodology has been known and followed for nearly two decades, the ways in which organizations have applied Agile have been as varied as the organizations themselves.

The Agile manifesto set out four basic values when it was first written in 2001:
- Individuals and interactions take precedence over processes and tools.
- Working software takes precedence over comprehensive documentation.
- Customer collaboration takes precedence over contract negotiation.
- Response to change takes precedence over mindlessly following plans.
Agile processes are applied across a range of development activities, from development of standalone applications to development of cloud-native business applications. The Agile Manifesto also contains several more principles that further explain what the values mean when put in practice.

These Agile principles can be condensed into a few areas of focus:

Prioritize the customer over the product

Agile development requires continuous customer engagement throughout the entire development process, with the end goal being frequent product delivery based on specific customer feedback. While this can present challenges for an Agile team, particularly when customers demand changes late in the development cycle, Agile maintains that the best way to elevate customer experience is to make the customer part of the development process.

Trust Agile Teams to Self-Organize and Self-Manage

Agile focuses on smaller teams that remain in constant communication with customers, as well as with each other. These teams should include both developers and those with knowledge of the larger business requirements being met through development. For these teams to work effectively, trust is key. Agile posits that self-organized teams, given leeway to pursue their projects in the way they see fit, result in the shortest development cycles.

Keep it Simple

Agile teams require simplicity. This correlates with Agile’s disdain for overuse of process. Agile seeks to minimize the total amount of work required, thus minimizing overall development time.

Agility Leads to Agility

Just as Agile teams continuously improve their product and the customer experience, they also constantly improve themselves. Constant evolution of teams and the processes they employ (remember, teams before processes) improves team agility itself.

While the Agile values and principles seem simple, effective application of the values and principles is another story entirely. Indeed, at times, the implementation of Agile development has violated the very first value of the manifesto by focusing on which tools and processes should be used. But, in an environment where more users are clamoring for more products and services, implementation of Agile methodologies is not just important, it is imperative.

Building on Agile with DevOps

The precise meaning of the term itself and the scope of DevOps remain a source of debate. At a high level, DevOps is the marriage of the development cycle with post-development activities, such as deployment and maintenance.

According to open source contributor Barbara Ericson of Cloud Defense, “Instead of IT operations and software development being siloed off from each other, DevOps breaks down the traditional boundaries that previously existed between them in order to achieve continuous integration and continuous delivery (CI/CD) of quality software features and applications to end users.”

Just as with Agile, the effective implementation of DevOps requires cultivating a DevOps culture within the organization. However, Agile and DevOps are also not mutually exclusive. That is to say, DevOps is not merely a replacement for or an evolution of Agile. Although Agile and DevOps differ in their structures and approaches, they can work in unison to achieve the ultimate goals of speedy deployment and enhanced customer experience.

While Agile and DevOps share common goals, they have not always agreed on how to achieve those goals. DevOps differs in many respects from Agile, but, at its best, DevOps applies Agile methodologies, along with lean manufacturing principles, to speed up software deployment.

One area of particular tension between Agile and DevOps is that the latter relies heavily on tools; in particular, when it comes to the automation of testing and deployment processes. But DevOps can overcome the resistance of Agile developers to tool usage simply by applying Agile principles themselves.

Effectively, DevOps proponents must convince Agile teams that dogmatic adherence to the underlying principles of Agile is actually inconsistent with Agile in the first place. Ironically, Agile developers who insist that the process is always bad actually violate Agile principles by refusing to acknowledge the benefits offered through change, another Agile principle.

The challenge is to have the Agile development teams trust in the automation efforts of DevOps, while at the same time encouraging the DevOps team to consider the business goals of deployment rather than pursuing speed of deployment for its own sake.

With constant communication between the Agile team and DevOps team (another Agile principle), development can achieve a degree of comfort with DevOps tasks and processes. This means that testing and deployment automation can proceed quickly, resulting in a decreased time to market, overall.

Ultimately, Agile and DevOps can work together most effectively when they recall their shared goals: smaller change increments based on feedback resulting in faster delivery cycles.

Continued Evolution: DevSecOps

As networks expand and businesses continue to integrate cloud-based services (in many cases, building multi-cloud environments), security issues become more and more complex. One area in which Agile is lacking is in integrating security into the development process. Unfortunately, in an Agile environment, application security is often an issue addressed after development, rather than as part of the process. Enter the next iteration of DevOps: DevSecOps.

DevSecOps revolves around three basic principles: ensuring data security while minimizing inconvenience for users in accessing data, using development tools that identify risks as early as possible in the development process and ensuring data encryption using technologies such as secure sockets layer (SSL) and virtual private networks (VPNs).

While DevSecOps and Agile may also disagree about the priority of tools, security must become an integral part of any development project. This is why Agile, DevOps and DevSecOps must all work together to ensure not only rapid deployment but secure deployment.

The Next Evolution

Agile and DevOps were both important evolutions of the software development process, but DevSecOps is the next evolutionary step. Despite a few disparate characteristics, Agile, DevOps and DevSecOps can and should work in unison to achieve their shared goal: the best possible customer experience and short deployment cycles.
December 24, 2021
Best of 2021 – The Next Evolutions of DevOps

As we close out 2021, we at devopsy.kinsta.cloud wanted to highlight the most popular articles of the year. Following is the seventh in our series of the Best of 2021.

As society continues to recover from the global COVID-19 pandemic and some semblance of normal starts to return, technology continues to move forward. The pandemic certainly changed what normal means and people, processes and technologies all have adapted and evolved to meet those challenges. The narrative that the pandemic caused of your organization’s digital transformation is true for many practitioners. With 2022 just around the corner, DevOps is on the cusp of maturation of previously bleeding-edge paradigms and a focus on engineering efficiency. Here are five DevOps evolutions you should keep an eye on going forward.

Engineering Efficiency is Front and Center

Engineering efficiency is a particularly broad umbrella term. At the foundational level, engineering efficiency focuses on making someone (for example, an engineer) more productive. There are a multitude of disciplines that intersect, from organizational design to engineering/developer experience. Engineering efficiency is increasingly critical for organizations today. The pandemic had two major impacts on the technology sector in terms of resources and staffing. The first, at the start of the pandemic, was the unprecedented and unpredictable availability of resources; physical locations were being shut down because of the medical severity of the pandemic and the “getting hit by a bus” factor was playing out in real life. Second, during the recovery period, the Great Resignation began, and with it, a fight for resources that has become increasingly intense. Both have meant that the availability of resources, especially ramped resources, is scarce.

The Spotify model of tribes/guilds of engineering resources is a promising evolution; folks can frequently move around, reducing the toil that engineering resources face and allowing for engineering resources to not only ramp up more quickly but also increasing engagement and retention. The industry continues to march toward standardization in areas that were typically bespoke, such as the release process, and I think we’ll see that continue to evolve, as well.

Git is Ubiquitous

Leveraging a source code management (SCM) solution used to be reserved for software engineers. But as the proliferation of “something-as-code” has touched multiple levels of the technology stack from networking, storage and, ultimately, a development pipeline, the march of operations-focused engineers adopting many of the traits of software engineers continues. Having iteration in your infrastructure stack with more ephemeral/disposable infrastructure is becoming the norm. Saving and packaging the multitude of configurations for the “something-as-code” source control is a natural area of evolution.

Building off of source control are the package managers—your Docker registries, Helm chart repositories, etc.—for more deployable artifacts; that said, getting those artifacts to production is a process. Because Git is becoming ubiquitous across the technology landscape, this leads to the continued adoption of GitOps. Since the seminal piece was written by Weaveworks in 2017 defining the GitOps paradigm, leveraging GitOps is seen as a viable paradigm for many organizations five years later, especially when starting with greenfield initiatives.

Kubernetes is No Longer Bleeding Edge

In 2022, Kubernetes will celebrate the eighth birthday of its first commits on GitHub. Taking a jog down technology memory lane, eight years before Kubernetes (in 2006), VMware was still a private company and it would be a few years before vSphere was even released (remember how it felt to run a workload on a virtual machine in 2014?). The Kubernetes ecosystem still moves quickly, but placing workloads on Kubernetes is not a novel concept anymore. As application infrastructure and architecture have adopted the Kubernetes way (in other words, being idempotent and ephemeral), there is maturity around running a suitable workload on Kubernetes.

Kubernetes by design is highly pluggable. If you do not like the opinion or implementation of something inside Kubernetes, you can replace that opinion. Don’t like how Kubernetes handles ingress traffic? Then choose from one of the many ingress controllers available today. That is just one example of dozens of pluggable areas. Because of this, Kubernetes is viewed as a dynamic, non-static resource. Using Kubernetes takes trial and error and getting your cluster to be robust, performant and reliable is an ongoing journey that requires ongoing iteration.

Authors Can Be the Enforcers

There is no question that in a modern software stack, we are inundated with data. The metrics soup and the continued science of and exploration of observability (metrics, traces, logs) provide us with a lot to work with. Making decisions, even automated ones, based on that data continues to be critical. With the rise of site reliability engineering (SRE) practices, reacting or anticipating spikes in usage, failures and even security-related items are par for the course for modern teams.

These decisions can be authored into policies that specific platforms understand, such as autoscaling rules for a cloud vendor or something like Open Policy Agent in the Kubernetes ecosystem. With so many items shifting left toward the development team, finding the right resource, skillset or institutional knowledge to author these rules can involve the input of several teams. Because of the rise of “something-as-code,” regardless of whether the author of these rules is a software engineer, DevOps engineer, platform engineer or anyone in between, that author has the ability to enforce.

Shifting Left … But Complexity Shifts Left, Too

The old adage is that complexity is like an abacus: You can shift complexity around, but it never really goes away. With the movement to shift responsibility left to development teams, this also means that associated complexity is shifting to the development teams. Modern platform engineering teams provide the infrastructure (compliant Kubernetes clusters) to teams and any workload that is run on those clusters is up to the development team that owns it. Typically, development teams then focus on features and functionality. Managing lots of non-functional requirements—and even core infrastructure requirements such as networking—can be a burden; think about how your organization would handle a service mesh.

If you are a DevOps or platform engineer, making your internal customers—your development teams—successful is a great goal to work toward. Crucial to this is disseminating expertise. This can be in the form of automation and education. A common practice with the DevSecOps movement is to have some sort of scanning step as part of the build or deployment process, disseminating the internals as far as how the scan is performed, what happens if something is found, etc. Gaining internal adoption is a journey and having a good developer experience built around clarity and stability are important.

Evolution in 2022

Even with all of the challenges, bumps and learnings that the last year has given us, technology continues to evolve and advance and even lower the barriers of entry to continually become more inclusive. Focusing on improving engineering efficiency and reducing toil will allow for more participation and lower the barriers to entry.

December 23, 2021
Only 30% of Orgs Fully Implement DevSecOps

With the pressure to release more rapidly, security is shifting left within the continuous development pipeline at most organizations. This imperative is increasing with the rise of cyberattacks. Yet both a lack of training and poor visibility are stalling many DevSecOps rollouts, a recent CSA study found.

DevSecOps is still a relatively new practice and has yet to reach maturity throughout most organizations. Though nearly all companies are on their way, only 30% of security professionals say they fully implement DevSecOps in practice today. That leaves most organizations in some state of DevSecOps ideation and planning.

CSA recently released its Secure DevOps and Misconfigurations 2021 report which was commissioned by Trend Micro. The report found that misconfigurations commonly arise from default security settings. A lack of internal guidance and accessibility of security resources commonly hinder DevSecOps maturity, too.

Below, I’ll review the other major takeaways from the report to see how they can inform organizations as they move to implement DevSecOps.

State of DevSecOps Adoption

DevSecOps is all about increasing the security around rapid deployment. It literally puts the security into DevOps by shifting security automation left and de-siloing the boundary between DevOps and IT security teams.

While most engineers realize the benefits of DevSecOps, the state of each organization’s DevSecOps journey is very different. While an impressive 90% of organizations are in some phase of the journey toward DevSecOps, only 30% are implementing DevSecOps while 24% are in the planning phase, 18% are designing and 18% are still refining their DevSecOps strategy. Just over one in ten security professionals say their team has no plans to invest in DevSecOps at all.

Looking to the future, 42% say they will implement full DevSecOps within the next 12 months. But due to its current nascent state, misconfigurations are still getting through. The primary reason cited for these misconfigurations was flawed or lacking internal guidance (33%).

Many professionals say there is simply not enough training, support or internal knowledge around vulnerabilities and misconfigurations. Other top reasons for misconfigurations include insecure default settings (18%) and negligence (16%).

Outside of misconfigurations, groups also report challenges with identity, authorization and access. Ensuring the correct privileges is vital to avoid escalation attacks, which are a prevalent issue. Between 2019 and 2020, 80% of companies experienced a data breach of some kind, many of which were due to misconfigured access controls. OWASP also reported that broken authorization is a top vulnerability for high-traffic web API endpoints.

Common Threat Mitigation Practices

To mitigate these threats, development and security teams must maintain best practices, yet certain issues prevent DevSecOps from emerging. For one, 60% of security professionals say their top challenge is insufficient visibility into security or compliance gaps—this is by far the most common challenge—and 11% also cite inconsistent cloud account onboarding. Finally, 10% say time-consuming, slow and/or insufficient architecture holds them back.

One way to mitigate threats is to introduce routine security reviews more frequently. Yet it’s hard to set a definite benchmark here, as the frequency at which organizations review their cloud infrastructure for vulnerabilities or misconfigurations varies widely. Currently, 22% perform such security reviews daily, 22% monthly, 18% weekly and 23% quarterly, according to the survey. This rate will likely increase as DevSecOps becomes more commonplace and automated within the development life cycle.

Staying up-to-date with modern security frameworks is another facet impacting DevSecOps adoption; and organizations tend to follow multiple frameworks to inform their security strategy. More than three-quarters (78%) follow the National Institute of Standards and Technology (NIST) guidelines, 67% follow the Center for Internet Security (CIS) benchmarks, 66% follow Cloud Security Alliance (CSA), 54% follow International Organization for Standardization (ISO) and 44% say they follow security recommendations from Amazon Web Services (AWS).

For example, NIST, the government-operated standards-setting body, recently set cybersecurity guidelines that recommended the use of zero-trust and service mesh across departments. These security guidelines can be thought of as architectural benchmarks, especially for large, highly-regulated industries.

DevSecOps Training Types

Finally, investing in community resources and training is another way to increase DevSecOps awareness. Only half of survey respondents reported their DevSecOps best practices resources were moderately accessible—thus, the onus is on leaders to democratize such knowledge within their organization.

The majority (81%) of respondents cited online articles and training as their primary form of learning about cloud security tools and vendors. Workshops, conferences and webinars follow closely. Organizations also adopt many internal knowledge-sharing methods in response to incidents. A full 85% said they conduct awareness training followed by table-top exercises (52%), attack simulations (45%) and protocol or response framework training (37%).

Future DevSecOps In The Cloud

It appears that most teams are still formulating their DevSecOps strategy. But looking to the future, 42% say they will implement full DevSecOps within the next 12 months. Simultaneously, more cloud traction is projected throughout the industry.

Right now, 40% of organizations have between 41% to 99% of their workloads in the public cloud. And 55% of organizations will have between 41% to 99% of their workloads in the public cloud in the coming year. The type of workloads will also likely evolve in the coming year, as more organizations move to container platforms, Function-as-a-service and other serverless capabilities. To fully reap the benefits these new technologies promise, organizations must undoubtedly respond to a new class of cloud-native vulnerabilities.

About The Report

The Cloud Security Alliance (CSA) is a not-for-profit organization with a mission to widely promote best practices for ensuring cybersecurity in cloud computing and IT technologies. The Secure DevOps and Misconfigurations survey was conducted from July 2021 through September 2021 and collected over 900 responses from a global pool of IT security professionals working in various sectors and organizational sizes. For a full copy of the report, you can swap some personal information for a PDF download here.

December 22, 2021
Rails 7 FTW | WFH is SOP | 586M Passwords Dumped

Welcome to The Long View—where we peruse the news of the week and strip it to the essentials. Let’s work out what really matters.

This week: Ruby on Rails 7.0 is go, working from home is still de rigueur, and HIBP gets far, far bigger.

DHH is Excited

First up this week: Ruby on Rails version 7.0 is now in production. It promises less bloat, better security and even more developer efficiency.

Analysis: Last chance for greatness

Talk to devs about Ruby on Rails and you’ll hear a wide range of opinion—all the way from “obsolete memory hog” to “productivity secret weapon.” But its important promise seems to be to encourage devs to use better design patterns.

Paul Krill: Ruby on Rails 7 rejiggers JavaScript support

Ruby on Rails 7, the latest version of the web application framework, is now available as a production release. … It gives developers a “no-Node” approach to front-end development while still providing access to NPM packages and modern JavaScript.
…
Most Rails applications will not require Node.js, given the new defaults. … Other improvements:
…
Encrypted attributes have been added to Active Record, enabling applications to offer at-work encryption in addition to traditional at-rest and in-transit. … With asynchronous query loading, two unrelated queries can be run concurrently. … The Spring application preloader is no longer on by default, as faster computers have made it unnecessary.

Let’s hear from Rails creator David Heinemeier Hansson—AKA DHH:

Seven is … the culmination of years of progress on five different fronts at once. … The part that really excites me … is how much closer it brings us to the ideal of The One Person Framework. A toolkit so powerful that it allows a single individual to create modern applications upon which [to] build a competitive business.
…
It’s been well over twenty years for me now, and a big release like this still makes me giddy. … What a glorious time to be working in web development.

What a whatnow? Here’s bb_matt:

I don’t pretend … it’s a “glorious time to be working in web development,” except for one important aspect: … Experienced programmers coming over from software development and starting to make amends [with] strong typing, robust CI/CD, a focus on TDD, a focus on domain modeling.
…
99% is still total **** though—but then again, so is most software.

Home Working Ain’t Gone Away

A month ago, my long-suffering editor permitted me to write, “WTH? We Wanna WFH.” And survey after survey says tech workers work better at home. After the last 21 months, there’s simply no denying it.

Analysis: Genie out of bottle—no going back

If you’re under the misapprehension that DevOps is somehow “special” and your workers will be glad to return—think again. Find a way to make it work, or work will stop—and your business will die.

Steven J. Vaughan-Nichols: How to lure employees back to the office? You can’t. Not now. Not ever.

Months have gone by, and the great resignation keeps rolling along. … People don’t want to catch COVID-19 [but] bosses still think they can force skilled workers to return to offices. … That’s not going to happen.
…
The problem? Many executives and owners haven’t gotten the clue yet. [But] any doubts you may have about people not doing a good job unless you’re looking over their shoulders should have vanished by now. … The monthly academic WFH research.com survey has found almost six out of 10 workers reported being more productive working from home. … In the Dice State of Remote Work report … 53% of technologists listed greater productivity as one of the main benefits.
…
It’s simple: It works better for them and for your company. … Anyone who’s been paying attention … must agree. You can either go along with the flow, or you can fight it and first lose your staffers and then your company

So what do DevOps managers think? One of them is kozikow:

100% remote is not great either. … When there was a long period of 100% remote during Covid, there were some issues. … We developed a policy that Mondays are required, Wednesday are encouraged, rest is work from home. … It was shared decision of the team.
…
I think that most people who are strongly against hybrid work just have to live in a miserable place like SF. … It’s unfortunate that so many software companies are in the Bay Area.

But you couldn’t pay some people enough to go back—people such as this Anonymous Coward:

No way in hell they’d ever be willing to pay me … to come back into an office. … No having to go out in ****ed up weather, no fighting traffic and … dangerous drivers, plus I get all that commuting time back.
…
And I don’t have to put up with Linda cooking fish in the microwave.

Have I Been Pwned? There’s a 38% Higher Chance

This week saw a huge bump in the size of Troy Hunt’s already-massive database of stolen credentials. In case you’ve not seen it, HaveIBeenPwned.com is an amazing free service that lets apps securely check a user’s password isn’t already used elsewhere on a compromised site.

Analysis: Build the API into your auth flow ASAP

The UK’s equivalent to the FBI has donated a huge trove of credentials—after deduping, it’s added more than 225 million new passwords to the database. Frankly, if you still allow your users to rely on any of the 847 million passwords in HIBP, it would be negligent.

Simon Sharwood: National Crime Agency finds 225 million previously unexposed passwords

Troy Hunt, of Have I Been Pwned (HIBP) fame, yesterday announced … the NCA shared 585,570,857 [creds] with HIBP, and Hunt said 225,665,425 … he hasn’t seen before in the 613 million credentials HIBP already stored. [HIBP] lets anyone conduct a secure search of stolen passwords to check if their credentials have been exposed.
…
The NCA’s statement to Hunt did not reveal the source of the password trove [but said they] “were able to identify a huge amount of … passwords in a compromised cloud storage facility. … It became clear that these credentials were an accumulation of breached datasets known and unknown.”
…
[The] release brings the total Pwned Passwords count to 847,223,402, a 38 percent increase.

How’s this relevant to DevOps? Troy Hunt explains:

In the last month, there were 1,260,000,000 occasions where a service somewhere checked a password against … HIBP’s Pwned Password API. … There are all sorts of amazing Pwned Passwords use cases out there. For example … checking their customers’ passwords on every registration, login or password change to see if it’s previously been breached.
…
The UK’s National Crime Agency has done some wonderful work over the years to combat cybercrime. … Today’s release is about turning on the firehose of new passwords and making them immediately available to everyone for free. Having this open to the community, owned by the community and supported by the FBI and NCA is an enormously pleasing result.

Time for a metaphorical thought experiment. LenKagetsu obliges:

You were told by a reliable source that an unknown person … posted a high-res image of … your house key … on the internet. Would you change the locks?

The Moral of the Story: Never switch horses in midstream

You have been reading The Long View by Richi Jennings. You can contact him at @RiCHi or tlv@richi.uk.

Image: David Emrich (via Unsplash)

December 21, 2021
Best of 2021 – Torvalds’ Bug Warning is a Lesson for Linux Users

As we close out 2021, we at devopsy.kinsta.cloud wanted to highlight the most popular articles of the year. Following is the third in our series of the Best of 2021.

Linux does, occasionally, raise security concerns. While many users see it as the most secure, robust and versatile operating system available — that’s this writer’s opinion, as well — security precautions still have to be taken.

A recent, widely publicized case illustrated this point; Linux creator himself, Linus Torvalds, warned against the use of the Linux 5.12 release. He described a “nasty bug,” and wrote that the situation is a “mess,” due to the use of swap files when adding Linux updates. This nasty bug, in fact, had the potential to destroy entire root directories.

Some of the main takeaways following this “mess” include: tread very carefully when installing early Linux releases, especially those that involve swapping files instead of partitions, and especially, despite Linux’s well-known security advantages, avoid becoming complacent, because Linux security is not always foolproof.

Hence, while the “state of Linux security today is quite good, and has evolved in a positive way with more visibility and security features built, like many operating systems, you must install, configure and manage it with security in mind; that is how cybercriminals take advantage, [via] the human touch,” said Joseph Carson, chief security scientist and advisory CISO at Thycotic, a provider of privileged access management (PAM) solutions.

A Patch for Nastiness

As Torvalds noted a few weeks ago, “most people don’t use a swap file, but a separate swap partition and the bug in question really only happens when you have a regular file system, and put a file on as a swap.”

“The bad news is that the reason we support swap files in the first place is that they do end up having some flexibility advantages, and so some people do use them for that reason. If so, do not use [release candidate] RC1,” Torvalds wrote. “Thus, the renaming of the tag.”

After issuing the warning, Torvalds released a patch that he says prevents the bug from destroying swap file systems. However, it may have already been too late for early adopters of release 5.12. Ubuntu, a leading Linux distro, can swap files by default.

“It is nasty bug if you are still using swap files,” Carson said. “If you do still use swap files, then you could be impacted, resulting in potential data loss or a corrupted system.”

DevOps teams – or anyone else running Linux and installing patches, whether on multi-servers or on individual workstations – still need, of course, to follow strict best practices. “Like any operating system, security depends entirely on how you use, configure or manage the operating system,” Carson said. “Each new Linux update tries to improve security; however, to get the value, you must enable and configure it correctly.”

Linux Goodness

The fact that Torvalds was so forthcoming about the bug, as well as the level of transparency that the Linux kernel offers, also demonstrates one of the many reasons Linux remains popular. Given that the Linux kernel, in one variety or another, is used “not only in about 50% of the internet servers of the world, but also in a substantial part of all our smartphones, it is good to see this level of transparency at ‘root level,” said Dirk Schrader, global vice president, security research at New Net Technologies (NNT), which providers cybersecurity and compliance software.

“The security of Linux is based on its transparency; the ability to review the code of a distribution,” says Schrader. “Quite often forgotten is that transparency also involves talking about the mistakes, the errors, those nasty bugs.”

Citing National Institute of Standards and Technology (NIST) vulnerability database statistics, Schrader described how, compared to the Windows family of desktop and server operating systems, for example, the Linux kernel shows better results for overall vulnerabilities. The number of vulnerabilities have also declined over the past four years, while Microsoft’s operating systems do not display the same trend, according to NIST’s national vulnerability database.

Since Linux’s famous kernel is open source and transparent, it is possible to extrapolate that there are a greater number of potential vulnerability watchdogs compared to those monitoring vulnerabilities in closed systems. Some may argue that Microsoft has been, at times, less successful at detecting vulnerabilities and issuing much-needed patches.

However, Linux users still must remain vigilant.

“Still, for any of the Linux distributions, anyone using the early release candidates — RC1 in particular — should make sure that their own development or build process is undergoing change control, so that no mishaps will transfer the nasty bug into a production environment,” said Schrader.

December 21, 2021
How Log4j Becomes a Serious DevOps Problem
The recent discovery of the Apache Log4j vulnerability has wide-ranging implications for anyone who develops software, especially for those in the DevOps realm. What’s most troubling about the vulnerability (CVE-2021-44228) is how prevalent the use of Log4j is. The vulnerability is reported in a vast array of applications and directly impacts numerous Apache projects, including Druid, Dubbo, Flink, Flume, Hadoop, Kafka, Solr, Spark and Struts. There are also numerous Cisco programs impacted by the vulnerability. The list of impacted applications and libraries seems to be almost infinite, extending to Elastic LogStash, GrayLog2, Neo4J, Steam, Twitter and even the VMware Tanzu family of programs.

Simply put, any application that uses Log4J for logging and tracing is subject to the identified family of attacks known as Log4Shell. Those attacks prove easy to exploit and can be used to take control of vulnerable servers.

Why DevOps Practitioners Should be Concerned

The world of DevOps is all about reiteration, where features and improvements to applications are slipstreamed into production environments. To accelerate development, the QA process is usually limited to the most recent changes.

In other words, developers, via their pipelines, can often establish a baseline of quality and then speed up their CI/CD pipelines by testing only what has changed since the last iteration of the application. This process ensures an acceptable level of quality without introducing what are perceived to be unnecessary steps.

However, those processes do not take into account newly discovered flaws in older APIs and libraries, meaning that those flaws could telegraph into deployed applications and leave DevOps teams completely unaware of the impact and possible disruption. Detecting flaws after an application is deployed is like putting the cart in front of the horse.

Further complicating the situation is that many developers lack a well-defined and well-maintained software bill of materials (SBOM), which means developers may be unaware of the components and/or libraries being used in a deployed application.

Without an SBOM, it becomes increasingly difficult to remediate an issue with an application—especially if developers are unaware of the issue to begin with.

Where Should Responsibility Lie?

More often than not, vulnerabilities introduced into applications become a game of finger-pointing, where developers blame library suppliers or infosec staffers, and infosec blames developers for not performing due diligence on the components they are selecting to incorporate into their applications.

Of course, finger-pointing only adds to the angst and delays the process of remediation and does nothing to determine where the responsibility for detecting and remediating vulnerabilities lies. DevOps teams need to put these squabbles aside and establish a process that deals with unintentionally introduced vulnerabilities. What’s more, DevOps teams need to invest in DevSecOps to make sure that vulnerabilities are detected and remediated as quickly as possible.

To avoid the potentially massive disruption that a vulnerability like Log4Shell can cause, those running DevOps should consider:
- Creating and maintaining an SBOM
- Deploying automated tools that scan for discovered vulnerabilities
- Automatically comparing the SBOM against vulnerabilities
- Including DevSecOps processes within their CI/CD pipelines
- Frequently testing applications for flaws
- Coordinating with cybersecurity personnel when a flaw is discovered
- Patching frequently
With more and more vulnerabilities discovered on an almost daily basis, DevOps teams must be ever vigilant about what components are incorporated into their applications. What’s more, with the increased use of low-code/no-code tools and integrated development environments (IDEs), DevOps will also need to consider if those tools use other libraries that can become compromised.
December 20, 2021
Bridging the AppSec and DevOps Disconnect

Research estimates that cybercrime is going to cost the world $10.5 trillion annually by 2025, so it is no surprise that cybersecurity has become a top priority for business leaders.

Today, security teams are striving to harden their systems against cyberattacks and improve their resilience through more employee security training, incident response programs and the implementation of technology to monitor and detect threats.

Many organizations are also adopting a shift left security strategy, where security is being addressed earlier in the application development life cycle. These organizations are realizing that finding security issues in their software after it’s released is much riskier and more costly. After all, the longer a bug is in the wild, the longer an attacker has to find and exploit it. By implementing a shift left security strategy, the window of opportunity for an attacker decreases or can be completely removed if a security issue is addressed in the development stages, while also offering organizations significant security benefits and cost savings.

Shifting left is a security testing strategy where DevOps and security teams work in tandem to identify and address vulnerabilities during early development stages. This process significantly improves security as it means vulnerabilities are identified and mitigated before they are introduced to the enterprise environment.

However, one of the biggest challenges organizations face when adopting a shift left security strategy (DevSecOps) is encouraging security teams and DevOps teams to work together.

Encouraging AppSec and DevOps Collaboration

Until recently, DevOps teams and AppSec teams were completely siloed. DevOps teams worked on accelerating the speed at which applications are released into the enterprise; application security teams only got involved in the later stages. However, this process has caused security concerns, particularly as more organizations move to agile development.

When security teams eventually got their hands on new applications and updates, security flaws were inevitably found that caused delays and friction with DevOps teams.

However, as organizations begin to embrace DevSecOps and shift left strategies, these two previously siloed teams are being forced to work together more closely. So, how can the friction be reduced and the disconnect bridged?

Culturally, some ingrained attitudes and behaviors challenge the success of any DevSecOps efforts. Security teams have seen DevOps processes accelerate the speed at which software is delivered, but without security considerations, while DevOps teams experienced security slowing down processes and giving inconsistent results and feedback on security issues. Each party has their own manager to please; their own set of metrics that they’re measured against and a priority list as long as their arms already. Both teams follow different processes and, crucially, use different tools. DevOps can’t get around the security tool complexity and lack of integration with their existing toolset and security teams have no control over the CI pipeline to best implement security assurance.

One of the best ways to overcome this friction is through better technology, process and culture that enables collaboration between teams. First, DevOps teams do care about security, but it might be lower on their priority list. Security teams must understand that DevOps teams care about code, quality and efficiency. They are not security experts and just because an organization is shifting left doesn’t mean they need to become one. Instead, organizations should implement tools for DevOps where security checks can be easily embedded.

So, what can organizations do to improve the collaboration between AppSec and DevOps teams to achieve DevSecOps?

DevSecOps is a culture and it requires a shift in thinking. Security teams and DevOps teams need to understand each other’s priorities and understand that by working together they are improving the overall security of the organization and decreasing the risk of cyberattacks. Does the DevOps team understand the potential risks a vulnerability could have on the organization overall? If not, security teams need to educate them.

Security automation tools offer great support when organizations are moving to DevSecOps, as they allow bugs and vulnerabilities to be flagged automatically as applications are being developed. This will allow them to be addressed before services go live; however, because they generate frequent alerts, this could add to DevOps’ already busy workloads. By deploying tools which can be synced with CI/CD pipelines and provide custom advisories so DevOps teams can address bugs themselves without having to ask AppSec teams, this can significantly speed up the process, reduce friction and ease workloads.

December 17, 2021
Log4j: It’s All About the Supply Chain, Baby!

In 2021, the security story in DevOps and DevSecOps has been the supply chain. So, it’s only fitting that we are currently experiencing the mother of all supply chain issues with the Log4j Log4Shell RCE vulnerability to close out the year.

I won’t waste your time rehashing what Log4j is, why it’s so dangerous and warning you to test all of your own and third-party apps to make sure you’re insulated against this. I will say, though, that I don’t think Log4j is a bug—I think it is a defect in design. Worse, as we are seeing, there are multiple defects in design. All of you Java haters out there—back in the box. Java is no worse or better than any other language that we regularly use to build applications.

So, what exactly is so special about supply chain security? Why do DevSecOps teams, developers and our security incident management teams have to interpret this as a warning and stay at the top of their game from now on? It’s simple. In the age of SaaSOps; in a world where we Frankenstein applications by cobbling and stitching together numerous third-party components (some open source, some not), we are increasingly reliant on code, even in our own applications, that we did not write. We probably didn’t test it, either. We rely more than ever on the “community” and on unknown third-party suppliers. It has always been a leap of faith with these apps and components, and this year some of those chickens have come home to roost.

The White House executive order on improving the nation’s cybersecurity calls for a software bill of materials (SBOM) for applications; that is, as they say at the tables in Las Vegas, a fine start, but we need more. If we really had a BOM included with the apps we use (and that includes SaaS apps), how many of you are actually going to read it and act on that information? And besides including all these components in a BOM, what further obligations do suppliers have to ensure that they are replacing and updating vulnerable code?

This issue becomes increasingly acute with open source. My partner and colleague Mitchell Ashley recently asked if too much open source was a bad thing. I don’t know if I would go that far; the more the better, as far as I am concerned. But we do need better monitoring and management of the open source code that’s out there. More testing before it goes GA, maybe? Yeah, the Cadillac open source projects might have the money and time to do that, maybe, but take Log4j as a perfect example: It’s two guys maintaining it in their free time and trying to do right by the community. How can they possibly keep up?

I used to think that the answer was in the repos. My friends at Sonatype had a component firewall that would not let you download an old or vulnerable version of code without warning. That was fine if you were downloading it for the first time. But if it was already embedded—then what? Can we scan for older components in an app? My friends at JFrog can do something similar and more with Xray. And Sonatype’s Nexus platform also allows current and prospective users to scan older apps for vulnerabilities—including scanning for Log4j, absolutely free. But is that enough?

All of these questions involve open source components. What about third-party SaaS apps? Often, we don’t even know what components or code are in those SaaS apps. Again, it is a leap of faith we are making when using them.

These supply chain security issues are a problem that we are just beginning to grasp. And while 2021 has been the year the stuff hit the fan and Log4j may be remembered as the biggest of the first, I, for one, am afraid to think about what else is coming.

I do know that the only way to tackle this and any future supply chain issues is if developers, DevSecOps, DevOps, SREs and our security incident response teams all work together. Just like most security issues, supply chain security is all of our responsibility. I hope Log4j hasn’t bitten you too hard, and that you are on top of your exposure. But let’s start thinking about the next one and how we can better prepare for it.

December 16, 2021