devopsy.kinsta.cloud

Category: DevOps Toolbox

Intel Expands Scope of OneAPI Toolkit Initiative

Intel has made available a 2022 edition of its oneAPI toolkit through which it makes processor extensions accessible to DevOps teams.

James Reinders, chief evangelist for Intel software products, said the goal is to make it simpler for DevOps teams to expose extensions to any company’s processors in the runtimes developers use.

At its core, the oneAPI toolkit is a collection of APIs that Intel created for x86, graphics processing units (GPUs), application-specific integrated chips (ASICs), field programmable gate arrays (FPGAs) and other classes of processors.

Reinders noted that developers have never had as much ready access to so many different processor classes before. The challenge organizations face, however, is making it simpler for developers to take advantage of those capabilities in a way that doesn’t require them to write low-level code, he added.

The oneAPI 2022 toolkit adds additional capabilities including the world’s first unified compiler implementing C++, SYCL and Fortran, data-parallel Python for CPUs and GPUs, advanced accelerator performance modeling and tuning and performance acceleration for artificial intelligence (AI) and ray tracing visualization workloads. Collectively, Intel said it has made 900 new and enhanced features available via the oneAPI toolkit in the last year, including deeper Microsoft Visual Studio Code integration, support for Microsoft Visual Studio 2022 and Microsoft WSL2 for Linux development on Windows.

As IT continues to evolve it will become more routine for developers to invoke heterogeneous classes of processors within the same application, said Reinders. Those capabilities will drive an explosion of software development innovation through the rest of the decade and beyond, he added.

Intel is especially focused on making it easier for developers to invoke inference engines that take advantage of processor extensions to run AI workloads, Reinders said. Going forward, most applications will take advantage of machine learning algorithms that run inference engines optimized by Intel via extensions to its processors. As those workloads are constructed, integrating AI models within DevOps workflows will become the next major challenge, noted Reinders.

It’s too early to say how simplified access to heterogeneous processors, also known as XPUs, might change the application development landscape. The one thing that is clear is IT organizations now have more options than ever. The issue now is finding a way to invoke all those capabilities in a way that doesn’t overwhelm developers. DevOps teams will play a key role in deciding which processors and associated extensions to make available to their development teams.

In the meantime, the battle for processor dominance will only intensify in the months ahead. DevOps teams will need to decide how comfortable they are using processors manufactured by a single company like Intel or whether they want to employ a mix of processors provided by multiple vendors. Regardless of what they decide, it appears APIs will play a critical role in making it easier to benefit from advances in processor technologies.

January 7, 2022
Theranos’s Holmes VC Reverb | 5G C-Band Marches On | Mozilla Jumps Shark

Welcome to The Long View—where we peruse the news of the week and strip it to the essentials. Let’s work out what really matters.

This week: The conviction of Theranos CEO Elizabeth Holmes leads to greater scrutiny of startups, higher-frequency 5G NR is creeping towards reality, and Mozilla gets criticized for promoting climate-harming scams.

1. Holmes Case Causes Valley Rethink?

First up this week: Elizabeth Holmes has been found guilty of defrauding Theranos investors. But how will the affair change Silicon Valley’s startup-and-funding culture?

Analysis: Don’t fake it—just make it.

This could be a seminal moment, causing investors to look more closely at startup founders and their outlandish promises. On the other hand, key Theranos investors—e.g., Murdoch, Cox, DeVoss—hardly fit the mold of Silicon Valley VCs.

David Streitfeld: The Epic Rise and Fall

F. Scott Fitzgerald’s creation … Jay Gatsby … was a bootlegger [and] sold fake bonds. Ms. Holmes chose Silicon Valley, the last and greatest of all human dreams. [She] was a natural salesperson, as good at bending reality as Steve Jobs himself.
…
She and her deputy and boyfriend, Ramesh Balwani … thought they could brazen it out in the Silicon Valley tradition until they had something that actually worked. [But] the verdict signaled the end of an era. In Silicon Valley, where the line between talk and achievement is often vague, there is finally a limit. … It is an epic rise and fall that will be chewed over in the coffee shops and juice bars of Palo Alto, Calif., until the tech industry departs [for] off-world colonies.
…
So where does this conviction leave … her investors and former fans? Ripe for the next huckster that comes along, probably. Some Silicon Valley promises are so sweet we just can’t get enough of them. Immortality. Crypto. Flying cars. Mars. Digital harmony. Wealth beyond compare.

Think different, thinks DeanonymizedCoward:

[There’s a] fine line between bombastic marketing spam, “fake it ’til you make it” pitches, and outright fraud. … The first is repugnant and sometimes unethical, but generally legal.
…
[But] If your product requires bull**** to get funded—even if it ultimately works—you’re a fraud. … Regulatory reforms … and a sort of corporate death penalty … can fix a lot of this by making it less financially practical to engage in ****ty behavior.

Who else is to blame? Ask DS999:

I am sure some of her investors didn’t really care whether it worked. They just wanted to get in early, ride the hype train up, and sell before actual results mattered.

2. FAA and 5G Carriers Kiss and Make Up

Higher-frequency 5G NR is creeping towards reality in the U.S., thanks to the FAA backing down. After the aviation regulator had vetoed C-band deployment—based on precious little evidence—it looks like as much as 280 MHz of extra spectrum is now ready to switch on.

Analysis: Nomadic DevOps gets more bandwidth

While band 77 is nowhere near the much-vaunted “FR2” extremely high frequency bands, this is a useful chunk of extra spectrum for mobile and nomadic apps. Next stop: ubiquitous Ka-band.

Jon Brodkin: FAA agrees not to seek any more 5G delays

The Federal Aviation Administration tentatively agreed not to seek any more 5G delays from AT&T and Verizon, potentially ending a battle over the aviation industry’s unproven claim that 5G … will interfere with airplane altimeters. [Their] C-Band spectrum licenses are for the frequencies from 3.7 GHz to 3.98 GHz.
…
The deal incorporates voluntary commitments that AT&T and Verizon previously made, including “C-Band radio exclusion zones” around airports for six months. [They] modeled the exclusion zones after those used in France, which are 910×2100 meters.
…
Assuming there are no further problems, AT&T and Verizon would be able to use their spectrum licenses without extra restrictions after July 5. … President Biden applauded the agreement … calling it “a significant step in the right direction.”

In summary? Here’s Dieter Bohn:

The agency in charge of airplanes is mad about wireless things. But the agency in charge of wireless things and the companies that make them lost a game of chicken.
…
GREAT JOB AMERICA. … I always said I wanted Internet Service Providers to be Dumb Pipes, but I never expected them to be this dumb.

Has anyone read the FAA’s detailed report? bburdge has:

I can see why the FAA is backing down here now, it only makes sense when your key report could be paraphrased as: “We got a couple grad students to run some basic model comparisons … using altimeter data from the aviation industry (with a proven track record of lying to us) and then went golfing for eight months before publishing … but didn’t bother … to actually test anything.”

3. Mozilla Forgets its Values

Firefox custodian, the Mozilla Foundation, has angered many by promoting a “grossly irresponsible” “Ponzi scheme” that harms the environment. What is this awful thing? I hear you ask. It’s cryptocurrency.

Analysis: Imaginary money is bad—is this the last straw for Mozilla?

Less than a year ago, Mozilla said it wants to be carbon-neutral, but it’s already reneged on that promise: Distributed proof-of-work systems require … well, work. And that work uses ridiculous amounts of energy per DeFi transaction. Electricity production is a zero-sum game, so cryptocurrencies and NFTs cause climate change by emitting ridiculous amounts of CO₂e.

Liam Proven: Mozilla founder blasts browser maker

A few days ago, Mozilla Foundation invited netizens on Twitter to send in cryptocurrency donations. … This move by the Firefox browser maker rapidly drew criticism.
…
As recently as September, Moz’s web-standards lead Tantek Çelik slammed cryptocurrencies as being unsustainable, stating: “[They] are harmful for sustainability. … Their energy requirements grow without any discernible upper bound, which is grossly irresponsible given the global environmental crisis.”
…
Mozilla has accepted crypto-coin donations for years, perhaps as part of its long-ongoing efforts to alienate its own users: It’s only the payment provider that changed.

Chief among critics was Jamie “jwz” Zawinski:

I founded Mozilla and I’m here to say **** you and **** this. Everyone involved in the project should be witheringly ashamed of this decision to partner with planet-incinerating Ponzi grifters.
…
Stop saying you were promised flying cars. Unless you were born in 1935, you weren’t promised flying cars, you were promised a cyberpunk corporate dystopia.

It’s a shark-jump moment thinks Jeremy Keith:

The most convincing argument I’ve heard yet for switching away from Firefox as my main browser comes from Mozilla. I wasn’t expecting that plot twist.

The Moral of the Story: What is safety to one is not always safety to another.

You have been reading The Long View by Richi Jennings. You can contact him at @RiCHi or tlv@richi.uk.

Image: Benjamin Hayward (via Unsplash)

January 6, 2022
LogDNA Survey Surfaces Observability ROI Concerns

A Harris Poll of 200 senior engineering professionals suggests it may be a while before most DevOps teams are able to achieve true observability in their IT environments.

The survey was conducted on behalf of LogDNA, a provider of an observability platform, and found nearly three-quarters of respondents (74%) are struggling to achieve their observability goals despite investing hundreds of thousands of dollars. In fact, well over a third (38%) said they are already investing $300,000 annually on observability tools.

Despite those investments, less than half of respondents are very satisfied with their ability to use log data. Key areas of frustration include the challenges of collaboration with colleagues spanning multiple teams (67%), the fact that the tools are not easy to use (66%) and the routing of security events (58%).

Nevertheless, the bulk of survey respondents said they are optimistic about observability. A full 85% of participants said they believe true observability is possible.

LogDNA CEO Tucker Callaway said the core issue that many organizations fail to recognize is that observability is really a data literacy and management challenge. Organizations are collecting logs, metrics and traces from a wide range of tools and platforms without thinking through optimal data pipeline construction. The result is an aggregation of data that has not been normalized to simplify surfacing actionable intelligence, he said. In fact, many organizations quickly discover that all they’ve really accomplished is increase their data storage costs, said Callaway.

In the absence of that data normalization, Callaway noted it becomes exceedingly difficult for cross-functional teams to query all the data being collected within a repository.

At its core, observability promised to make it easier to identify IT issues before they cause a disruption. Observability in one form or another has always been a core tenet of any DevOps best practice. Initially, DevOps teams focused on continuous monitoring as the most effective way to proactively manage application environments. However, it can still take days, sometimes weeks, to discover the root cause of an issue. Observability platforms make it possible to correlate events in a way that makes it easier for analytics tools to identify anomalous behavior that might be indicative of the root cause of an IT issue.

In contrast, traditional IT monitoring focuses on predefined metrics to identify when a specific platform or application is performing within expectations. Observability platforms combine metrics, logs and traces—a specialized form of logging—to make it easier for IT professionals to interrogate data being generated by a wide range of DevOps tools and platforms.

Of course, the assumption is that DevOps teams actually know what queries to launch to determine what the root cause of an IT issue might be. In the longer term, machine learning algorithms should make it easier to identify anomalies within IT environments. In the meantime, however, achieving observability still requires a fair amount of DevOps expertise that is not always readily available.

Eventually, observability will become much more automated than it generally is today. As that transition occurs, IT teams should find the ROI of observability is steadily improving.

January 4, 2022
The Future of OpenTelemetry for Observability

The observability landscape is changing. Today’s digital transformation journey has highlighted the need for observability as organizations look to capture, analyze and correlate all data across every level of the application stack including logs, metrics, traces and events. With the rampant rise of data, collecting these signals is imperative to both business insights and system optimization.

Over the last several years, open source data collection has given software and engineering teams the freedom to decide which technologies they want to standardize on for data collection and which vendors they would like data sent to. While open source data collection brought a lot of freedom and advantages, it also created some new problems. The lack of a single, comprehensive framework for all the critical signals observability tracked created a lot of friction for developers. As we move into the new year, businesses are taking the opportunity to reflect on how they can better use their data and apply intelligence to help customers make decisions and sift through all the noise. Enter OpenTelemetry.

What is OpenTelemetry?

OpenTelemetry is a collection of tools, APIs and SDKs that instrument, generate, collect and export all machine data; it was born from the need to develop an industry-wide standard for software instrumentation. By leveraging a single cloud-native framework to complete observability, OpenTelemetry is becoming the gold standard for collecting machine data.

OpenTelemetry consists of three primary components: Specification, collector and instrumentation. The specification uses an API, SDK and the semantic conventions and protocol of OpenTelemetry to determine the cross-language requirements for all implementations. The collector removes the need for multiple data collection agents. It is a vendor-neutral method for receiving, processing, transforming and exporting data. The instrumentation libraries allow users to integrate their applications with OpenTelemetry using auto-instrumentation and support from multiple frameworks and languages. Together they provide a robust framework for companies to disrupt their existing observability practices and amplify them even further.

It’s All About the Customers

Most modern enterprises have complex distributed microservices consisting of tens, hundreds or thousands of components that are all working together to deliver on customer experience. If organizations can’t correlate between those components or the different signals easily, it will slow down a business’s ability to effectively monitor, diagnose and troubleshoot issues.

For years, organizations have leaned into observability by relying on multiple infrastructure vendors to gather, enrich and ship data to vendors. Still, many still do not maximize observability’s most pronounced ability: Correlating complex systems and signals.

As end users desire deep insight into their data, they also expect vendors to offer the best experience to identify and resolve issues within their systems with the help of observability. With true observability, organizations can give customers back their time to focus on what matters, which is managing the digital experiences for their customers and empowering them to meet their customers’ needs.

A Similar Road to Adoption

Much like Kubernetes has been essential for the growing adoption of software containers, the industry is already witnessing the same adoption approach around OpenTelemetry for observability. Kubernetes won the war for becoming the standard for container orchestration. Now, OpenTelemetry will bring the same standards needed for observability with comprehensive visibility into the health and performance of every layer of the stack.

OpenTelemetry has become widely adopted by the software development community, but it’s still at the beginning of its journey. The developer community made Kubernetes a success, and we will see the same with OpenTelemetry as customers demand better integration and less lock-in. For vendors involved around applications and software—from more traditional approaches like application performance management to software development pipelines and management tools and through to cloud-native services—supporting OpenTelemetry will be table stakes in the future.

With extensive portability, greater developer control and strong vendor and cloud provider support, OpenTelemetry has opened a new world to customers by helping to improve achievable results across the application infrastructure, from the software development process to areas like security.

How can businesses continue to move the needle on observability while balancing growing customer demands? Quite simply: If organizations aren’t thinking about OpenTelemetry now, they are losing competitive advantage by lacking the proper visibility to deliver on the service level objectives that customers expect in this day and age.

January 3, 2022
Best of 2021 – How to Revoke JSON Web Tokens (JWTs)
As we close out 2021, we at devopsy.kinsta.cloud wanted to highlight the most popular articles of the year. Following is the seventeenth in our series of the Best of 2021.

One of the most common questions about JSON Web Tokens (JWTs): Once they’re issued, how can they be revoked?

What Are JWTs?

JSON Web Tokens are portable, industry-standard identity tokens. They are issued after a login request by a central identity server and used to identify and credential a user and grant access to resources. They can be presented by clients such as browsers and external programs. Applications and APIs can examine them to ensure the caller is properly authenticated and authorized.

The Challenges Involved

One of the most common questions is how to revoke them. The problem is that once they are issued, no further communication is needed with the identity server. That’s one major benefit of JWTs decentralization.

Let’s say you use RSA public/private key signing for secure data transmission. After the IdP signs a JWT using the private key, any service that has the public key can verify the integrity of the token.

Let’s use the Todo-Backend API as an example. The architecture might look something like this:

Now, Todo-Backend can use JWT and the public key to verify the user’s ID and other attributes, and then use that ID to perform operations on that user’s data. Once the JWT has been issued, Todo-Backend doesn’t communicate with the identity provider at all. As a result, if an administrator has locked or deleted that user’s account in the identity provider, Todo-Backend won’t know about it.

Managing Revocations Using a Distributed Event System

The most common way to revoke access to resources protected by a JWT involves setting its duration to a short period of time and revoking the refresh token so that the user can’t generate a new token. This does not revoke the JWT per se; it does solve the root issue, which is to limit access.

In other words, you can set the JWT’s expiration duration to a short period (e.g., anywhere from a few seconds to, say, ten minutes) and set the refresh token’s expiration duration to a longer period (e.g., a two-week or two-month window).

During the normal flow, the Todo API would accept the JWT until it expires, at which time access would be denied. At that time the client could present the refresh token to the identity provider, which would send a new JWT to the client. That new JWT would be valid for another ten minutes.

An administrator can, at the identity provider, revoke the refresh token at any time. Any subsequent request for a new JWT by a client holding that refresh token would fail and access to the Todo API would be lost.

However, while ten minutes isn’t long, this timeframe still comes with risks.

One strategic way to handle this challenge is to set up a distributed event system that notifies services when refresh tokens have been revoked. Upon receiving this event broadcast, backends/services would update a local cache that lists users with revoked refresh tokens. After a JWT is verified, this cache would then be checked to determine whether or not access should be revoked.

This distributed event method truly revokes the JWT, rather than simply waiting for it to expire, as mentioned above.

An Example of Revoking JWTs

Here’s a sample solution that provides support for events and webhooks that can be used to implement this revocation strategy. In this example, when a user logs out or has their refresh token otherwise revoked, a jwt.refresh-token.revoke event is emitted. This includes the user ID and the application for which the revocation occurred. It also includes the JWT lifespan for this application (in the below case, 600 seconds). Any JWT for this user that was issued before 600 seconds from the point in time when this event was received is thus revoked.

For example:
- User logs in, and JWT ‘abc’ is issued at 1:11pm. JWT is good until 1:21
- User logs out at 1:15. Entry is added stating that any JWT for this application and user expiring before 1:25 is revoked.
- Application tries to use JWT ‘abc’ at 1:16. Todo-Backend does not deliver data because this JWT is on the denylist.
- User logs in, and a new JWT ‘def’ is issued at 1:18. JWT is good until 1:28.
- Application tries to use JWT ‘def’ at 1:19. Todo-Backend delivers data because this JWT is good until 1:28 and is therefore not on the denylist.
What you need to do is write a simple webhook that will receive this event and inform JWTManager that the refresh token for applicationId for this user has been revoked.

JWTManager maintains a list of user IDs whose JWTs are revoked. You could also use JWT IDs (the `jti` claim), which, in some cases, might be simpler. JWTManager also launches a thread to clean up the cache to remove expired users and avoid running out of memory.

For each API call, in addition to all the other checks that should be performed against a JWT (verifying the signature, expiration, issuer and so on), the backend needs to check validity with JWTManager.

As a final step, configure the webhook:

The Revocation Process, Simplified

Using the process described above, you can revoke a user’s refresh token and broadcast the event using a webhook. The webhook receivers then update JWTManager, which will render JWTs for that user unusable.

The good news is that you can use a webhook and event system to build this feature into your application quickly. You can also write JWTManager implementations into Java, Node and other client libraries for future use. Other languages, such as Ruby, have denylist implementations already written.

Ultimately, all that’s required are the use of refresh tokens, an API that enables revocation of refresh tokens and a system for distributing the revocation event. This solution should work well with most systems, even large systems with numerous backends.
December 28, 2021
Best of 2021 – Transform Legacy Java Apps to Microservices

As we close out 2021, we at devopsy.kinsta.cloud wanted to highlight the most popular articles of the year. Following is the sixteenth in our series of the Best of 2021.

Velocity is one of the major tenets of DevOps, and anything that maintains or improves velocity benefits the overall process. Yet, there are still many speed bumps in the DevOps process that threaten to derail progress and slow DevOps processes to a crawl. One of the biggest speed bumps arises when transforming legacy monolithic applications into something more cloud-friendly—namely, microservices.

Microservices are clearly the future of applications developed for the cloud; research giant IDC predicts that 90% of all new apps deployed by 2022 will feature microservices architectures. IDC also notes the primary benefits of microservices are the improvements to developers’ ability to design, debug, update and leverage third-party code. Although DevOps tends to be about developing internal code, the benefits offered by a microservices architecture remain much the same.

Microservices: Getting There

Moving toward a microservices architecture can be straightforward or complex, depending on the path an organization is taking and where they begin. For example, if the goal is to transform legacy applications into microservices, the process can be complex and tedious. Transformation of legacy applications often means manually disassembling legacy code and breaking it up into standalone code elements that will run as microservices. That greatly complicates things for the DevOps pipeline, where fast iteration of existing code is the norm, not the exception.

However, there are tools coming to market that can ease the process. One such tool is vFunction, which recently came out of stealth with the mission to transform monolithic Java apps into microservices.

vFunction has an interesting approach to that process. The vFunction platform is deployed locally, where the vFunction Java virtual machine (JVM) agent examines legacy applications and gathers data which is then collected on the vFunction server. From there, software architects can further investigate the collected information about the legacy applications, and start designing a microservices architecture to modernize the codebase. “Once developers are satisfied with the design, vFunction generates a service specification file, which contains all the information required to create the microservice,” said vFunction CEO and co-founder Moti Rafalin. “That file is used by our automation engine, which scans the original code and copies the relevant artifacts to create the new service.”

Rafalin said that vFunction leverages supervised learning techniques, graph theory and clustering algorithms to identify dead code and code anomalies that could prevent a clean breakdown, or “decomposition,” of the legacy application. Ultimately, vFunction has the potential to automate many of the manual steps that DevOps engineers used to perform to break monolithic applications into microservices. According to Rafalin, vFunction makes the process as much as 15 times faster and can achieve somewhere between 80% to 95% automation. Currently, vFunction is focused solely on Java, which it deems the biggest opportunity today. However, the company has plans to support applications written in other languages in the future.

Beyond newcomer vFunction, there are other ways to deal with moving legacy Java applications into the cloud. One example is Anthos, which is a managed application platform that extends Google Cloud services into DevOps environments. Another is IBM’s Mono2Micro, which is an AI-based, semi-automated toolset to assist with refactoring monolithic Java applications.

For those looking to build microservices-based applications from scratch, numerous tools exist. Most interesting are the development environments that use rapid application development (RAD) techniques and leverage low-code models. There are dozens of RAD platforms available; prime examples being WaveMaker , OutSystems and AgilePoint.

In most cases, those moving legacy applications to a microservices model will usually select numerous tools, based on the particular use case. For some, it may be easier to rewrite smaller legacy applications; for others, refactoring may be the only way to capture the intellectual property in legacy applications. Either way, carefully selecting the appropriate tools goes a long way towards maintaining velocity in the DevOps realm. Maintaining that velocity proves critical for building cloud-friendly applications on budget and on time.

“Unless one modernizes those monolithic applications to be cloud-native, they end up requiring very large machines and often end up paying more to the cloud providers,” Rafalin said. “Organizations now understand that, if they want the true benefits of the cloud, they need to modernize these apps. And this starts with adopting an architecture that includes microservices, APIs, and modern design principles.”

The lesson is to choose wisely, and select toolsets that support scale as well as growth, and that can support the requirements of microservices.

December 27, 2021
Best of 2021 – Nine Pillars of DevOps Best Practices

As we close out 2021, we at devopsy.kinsta.cloud wanted to highlight the most popular articles of the year. Following is the fourteenth in our series of the Best of 2021.

In a prior blog I explained my 27-factor DevOps assessment model which includes nine pillars of DevOps and three dimensions of people, process and technology.

The nine pillars represent categories of practices. In this blog, I include examples of practices for each of the nine pillars. In a future blog I will explain how to use these practices to conduct a gap assessment for your DevOps practices.

Leadership Practices for DevOps

• Leaders demonstrate a long-term vision for organizational direction and team direction.
• Leaders intellectually stimulate the team by encouraging them to ask new questions and question basic assumptions about the work.
• Leaders provide inspirational communication that inspires pride in being part of the team, says positive things about the team, inspires passion and motivation and encourages people to see that change brings opportunities.
• Leaders demonstrate support by considering others’ personal feelings before acting, being thoughtful of others’ personal needs and caring about individuals’ interests.
• Leaders promote personal recognition by commending teams for better-than-average work, acknowledging improvements in the quality of work and personally complimenting individuals’ outstanding work.

Collaborative Culture Practices for DevOps

• The culture encourages cross-functional collaboration and shared responsibilities and avoids silos between Dev, Ops and QA.
• The culture encourages learning from failures and cooperation between departments.
• Communication flows fluidly across the end-to-end cross-functional team using collaboration tools where appropriate (for example Slack, HipChat, Yammer).
• The DevOps system is created by an expert team, and reviewed by a coalition of stakeholders including Dev, Ops and QA.
• Changes to end-to-end DevOps workflows are led by an expert team, and reviewed by a coalition of stakeholders including Dev, Ops and QA.
• DevOps system changes follow a phased process to ensure the changes do not disturb the current DevOps operation. Examples of implementation phases include: proof of concept (POC) phase in a test environment, limited production and deployment to all live environments.
• Key performance indicators (KPIs) are set and monitored by the entire team to validate the performance of the end-to-end DevOps pipeline, always. KPIs include the time for a new change to be deployed, the frequency of deliveries and the number of times changes fail to pass the tests for any stage in the DevOps pipeline.

Design-for-DevOps Practices for DevOps

• Products are architected to support modular independent packaging, testing and releases. In other words, the product itself is partitioned into modules with minimal dependencies between modules. In this way, the modules can be built, tested and released without requiring the entire product to be built, tested and released all at once.
• Applications are architected as modular, immutable microservices ready for deployment in cloud infrastructures in accordance with the tenets of 12-factor apps, rather than monolithic, mutable architectures.
• Software source code changes are pre-checked with static analysis tools, prior to commit to the integration branch. Static analysis tools are used to ensure the modified source code does not introduce critical software faults such as memory leaks, uninitialized variables, and array-boundary problems.
• Software code changes are pre-checked using peer code reviews prior to commit to the integration/trunk branch.
• Software code changes are pre-checked with dynamic analysis tests prior to committing to the integration/trunk branch to ensure the software performance has not degraded.
• Software changes are integrated in a private environment, together with the most recent integration branch version, and tested using functional testing prior to committing the software changes to the integration/trunk branch.
• Software features are tagged with software switches (i.e., feature tags or toggles) during check-in to enable selective feature-level testing, promotion and reverts.
• Automated test cases are checked in to the integration branch at the same time code changes are checked in, together with evidence that the tests passed in a pre-flight test environment.
• Developers commit their code changes regularly, at least once per day.

Continuous Integration Practices for DevOps

• A software version management (SVM) system is used to manage all source code changes. (Git, Perforce, Mercurial, etc.)
• A software version management (SVM) system is used to manage all versions of code images changes used by the build process. (Git, Perforce, Mercurial, etc.)
• A software version management (SVM) system is used to manage all versions of tools and infrastructure configurations and tests that are used in the build process. (Git, Perforce, Mercurial, etc.)
• All production software changes are maintained in a single trunk or integration branch of the code.
• The software version(s) for supporting each customer release are maintained in a separate release branch to support software updated for each release.
• Every software commit automatically triggers a build process for all components of the module that has code changed by the commit. The system is engineered such that resources are always sufficient to execute a build.
• Once triggered, the software build process is fully automated and produces build artifacts, provided the build time checks are successful.
• The automated build process checks include unit tests.
• Resources for builds are available on-demand and never block a build.
• CI builds are fast enough to complete incremental builds in less than an hour.
• The build process and resources for builds scale up and down automatically according to the complexity of the change. If a full build is required, the CI system automatically scales horizontally to ensure the builds are completed as quickly as possible.

Continuous Testing Practices for DevOps

• Development changes are pre-flight tested in a clone of the production environment prior to being integrated to the trunk branch. (Note: “production environment” means “variations of customer configurations of a product.”)
• New unit and functional regression tests that are necessary to test a software change are created together with the code and integrated into the trunk branch at the same time the code is. The new tests are then used to test the code after integration.
• A test script standard is used to guide test script creation, to ensure the scripts are performing the intended test purpose and are maintainable.
• Tests are selected automatically according to the specific software changes. CT is orchestrated dynamically, whereby the execution of portions of the CT test suites may be accelerated, or skipped entirely, depending on how complex or risky the software changes are.
• Test resources are scaled automatically according to the resource requirements of specific tests selected and the available time for testing.
• Release regression tests are automated. At least 85% of the tests are fully automated and the remaining are auto-assisted if portions must be performed manually.
• Release performance tests are automated to verify that no unacceptable degradations are released.
• Blue/green testing methods are used to verify deployments in a staging environment before activating the environment to live. A/B testing methods are used together with feature toggles to try different versions of code with customers in separate live environments. Canary testing methods are used to try new code versions on selected live environments.
• The entire testing life cycle, which may include pre-flight, integration, regression, performance and release acceptance tests are automatically orchestrated across the DevOps pipeline. The test suites for each phase include a predefined set of tests that may be selected automatically according to predefined criteria.

Elastic Infrastructure Practices for DevOps

• The data and executable files needed for building and testing builds are automatically archived frequently and can be reinstated on demand. Archives include all release and integration repositories. If an older version of a build needs to be updated, then the environment for building and testing that version can be retrieved and reinstated on demand and can be accomplished in a short time (for example, minutes to hours.)
• Build and test processes are flexible enough to automatically handle a wide variety of exceptions gracefully. If the build or test process for a component is unable to complete, then the process for that failed component is reported and automatically scheduled for analysis, but build and test processes for other components continue. The reasons for the component failure are automatically analyzed and rescheduled if the reason for the failure can be corrected by the system; if not, then it is reported and suspended.
• System configuration management and system inventory is stored and maintained in a configuration management database (CMDB).
• Infrastructure changes are managed and automated using configuration management tools that assure idempotency.
• Automated tools are used to support immutable infrastructure deployments.
• Equal performance for all. The user performance experience of the build and test processes by different teams are consistent for all users, independent of location or other factors. There are SLAs and monitoring tools that ensure the user performance experience is consistent for all users.
• Fault recovery mechanisms are provided. Build and test system fault monitoring, fault detection, system and data monitoring and recovery mechanisms exist. They are automated and are consistently verified through simulated failure conditions.
• Infrastructure failure modes are frequently tested.
• Disaster recovery procedures are automated.

Continuous Monitoring Practices for DevOps

• Logging and proactive alert systems make it easy to detect and correct DevOps system failures. Logs and proactive system alerts are in place for most DevOps component failures, and are organized in a manner to quickly identify the highest-priority problems.
• Snapshot and trend results of each metric from each DevOps pipeline stage (for example, builds, artifacts, tests) are automatically calculated in process and visible to everyone in the Dev, QA and Ops Teams.
• Key performance indicators (KPIs) for the DevOps infrastructure components are automatically gathered, calculated and made visible to anyone on the team that subscribes to them. Example metrics are availability (uptime) of computing resources for CI, CT and CD processes, time to complete builds, time to complete tests, number of commits that fail and number of changes that need to be reverted due to serious failures.
• Metrics and thresholds for DevOps infrastructure components are automatically gathered, calculated and made visible to anyone on the team that subscribes to them. Example metrics are availability (uptime) of computing resources for CI, CT and CD processes, time to complete builds, time to complete tests, number of commits that fail and number of changes that need to be reverted due to serious failures.
• Process analytics are used to monitor and improve the integration, test and release process. Descriptive build and test analytics drive process improvements.
• Predictive analytics are used to dynamically adjust DevOps pipeline configurations. For analysis of test results, data may indicate a need to concentrate more testing in areas that have a higher failure trend.

Continuous Security Practices for DevOps

• Developers are empowered and trained to take personal responsibility for security.
• Security assurance automation and security monitoring practices are embraced by the organization.
• All information security platforms that are in use expose full functionality via APIs for automation capability.
• Proven version control practices and tools are used for all application software, scripts, templates and blueprints that are used in DevOps environments.
• Immutable infrastructure mindsets are adopted to ensure production systems are locked down.
• Security controls are automated so as not to impede DevOps agility.
• Security tools are integrated into the CI/CD pipeline.
• Source code for key intellectual property on build or test machines are only accessible by trusted users with verified credentials. Build and test scripts do not contain credentials for access to any system that has intellectual property. Intellectual Property is divided such that not all of it exists on the same archive and each archive has different credentials.

Continuous Delivery Practices for DevOps

• Delivery and deployment stages are separate. The delivery stage precedes the deployment pipeline.
• All deliverables that pass the delivery metrics are packaged and prepared for deployment using containers.
• Deliverable packages include sufficient configuration and test data to validate each deployment. Configuration management tools are used to manage configuration information.
• Deliverables from the delivery pipeline are automatically pushed to the deployment pipeline, once acceptable delivery measures are achieved.
• Deployment decisions are determined according to predetermined metrics. The entire deployment process may take hours, but usually less than a day.
• Deployments to production environments are staged such that failed deployments can be detected early and impact to customers isolated quickly.
• Deployments are arranged with automated recovery and self-healing capabilities in case a deployment fails.

What This Means

DevOps is a powerful tool that enables many benefits for organizations that use it. Achieving performance efficiently with DevOps depends on following best practices. By following the nine pillars of practices enumerated in this blog, organizations can achieve the performance potential that DevOps has to offer.

December 26, 2021
Best of 2021 – 11 Open Source DevOps Tools We Love For 2021

As we close out 2021, we at devopsy.kinsta.cloud wanted to highlight the most popular articles of the year. Following is the eighth in our series of the Best of 2021.

DevOps isn’t just a cultural shift — it requires great tools to come to fruition. Below, we’ve pulled together a list of some of the most well-loved DevOps tools available today. But, throwing loads of money into fancy SaaS solutions can quickly gobble up the cloud budget. These DevOps tools all are open source, and enable everything from container builds and orchestration to microservices networking, configuration management, CI/CD automation, full-stack monitoring and more. Here are some of our favorite open source DevOps tools for 2021.

1. Kubernetes

With the ubiquity of microservices and container-based software, it’s no surprise that Kubernetes tops this year’s list of open source DevOps tools. Kubernetes, adoption of which rose by 48% in 2020, is used to orchestrate containers. Instead of releasing microservices manually, Kubernetes can automate deployment, maintenance and scaling of groups of containers in production. Kubernetes, sometimes written as K8s, is hosted by the Cloud-Native Computing Foundation (CNCF).

2. Docker

Docker, the software, is a free and open source platform used to build, ship and run an application as a lightweight container. Containers package up the binaries, libraries, configuration files and dependencies required for a program to run. Over the past decade, containers have played a key role in agile development, and Docker containers led the revolution. At its core is the Docker Engine. Docker Hub is also an excellent resource for finding and sharing prepackaged functions as containers. Also, to plug container vulnerabilities, it may be helpful to use open source container auditing tools like Docker Bench or Anchore.

3. Istio

Microservices are a handy development style, yet they bring new development and architecture concerns. Namely, how do we apply networking policies like security, encryption, observability and telemetry elements consistently across all our services? Well, service mesh is one answer. Service mesh places a sidecar proxy next to each container and abstracts these networking capabilities to a control plane. Istio is one such open source service mesh that has seen widespread adoption. Istio is built on Envoy, opening it up to plugins and extensibility options. We should also mention Linkerd and Kuma as viable open source service mesh alternatives.

4. GitHub Actions

GitHub is arguably the most popular source control and software collaboration platform on the planet. The GitHub platform itself, based on Git, has seen some significant updates in the past few years. Most notable is the GitHub Actions capability. GitHub Actions enable software packages hosted on GitHub to accept inputs and trigger other processes. This could help automate some cool DevOps workflows within GitHub, such as code reviews, branch management or CI/CD processes — the combinations of possibilities here are endless. GitHub Actions are, essentially, YAML files hosted in GitHub repositories that leverage GitHub webhooks. Though this is more of a feature than an open-source tool, we feel it’s important to include here. Actions is free for public repositories with a limit of 100 actions.

5. Jenkins

A big part of the DevOps philosophy is finding ways to automate and deploy new iterations more efficiently. Part of this goal is creating a streamlined continuous integration and continuous delivery (CI/CD) pipeline. Jenkins is an open source automation server with hundreds of plugins to automate the building, deployment and testing of software projects. Although GitHub Actions could theoretically replace a CI server in the future, CI tools like Jenkins, CircleCI, TravisCI and GitLab Community Edition still are preferred by many DevOps teams.

6. Prometheus

Metrics and alerting systems are crucial for site reliability engineers to visualize applications and react to issues. Prometheus, a graduated CNCF project, is a well-loved open source monitoring solution. A Prometheus server collects time-series metrics by scraping HTTP endpoints and generates a system to interact with this data, offering deep querying, visualization, storage and other capabilities. Check out this Awesome Prometheus list for Prometheus introductions and additional resources.

7. Ansible

Ansible is all about automation. Ansible, an open source project sponsored by Red Hat, can be used to automate things like cloud provisioning, networking, deployment, configuration management and other tasks. Ansible has a simple yet effective architecture that is relatively easy to assemble — all you need is a text editor and command line. You describe your infrastructure in a text document and organize your desired states in a playbook. For an example in practice, see how OpenIO uses Ansible. “Ansible is our standard tool not only to deploy the OpenIO core, but also our WebUI, OIO-FS and all upcoming options,” writes Cédric Delgehier, Ops at OpenIO.

8. Chef

Chef is another infrastructure-as-code (IaC) solution for automating your configuration management. Chef uses Ruby to automate server configurations and works well with all major cloud service providers (CSPs). This can be very useful when creating and provisioning large quantities of machines. Like other automation tools on this list, the user describes their components and their states in a declarative format. In Chef, these are known as “recipes,” which can be grouped together into “cookbooks.” You can’t diss Chef for not being on-theme!

9. Terraform

Terraform is another IaC tool that can be used for initiation of building, versioning and further automation using configuration files. “Terraform is a tool for building, changing and versioning infrastructure safely and efficiently,” as it is described on GitHub. Terraform follows an “execution plan” that a user creates with high-level syntax. One unique aspect of Terraform is its emphasis on versioning — this allows you to version the blueprint of your service just as you would your software.

10. JAMStack

As I’ve covered before, JAMStack combines JavaScript, APIs and markdown for constructing web-based applications. While more of a “headless development” methodology than a single open source tool, JAMStack projects are often built using open source components. For example, JAMStack often leverages open source headless content management systems such as Ghost, Strapi and/or Netlify CMS.

11. ELK Stack

ELK Stack is the union of three open source projects maintained by Elastic: Elasticsearch, Logstash and Kibana. With these three components, developers can take in and log data from any source and create helpful visualizations. This centralized logging is enabled with a NoSQL database for storage with Elasticsearch, processing and data collection with Logstash and visualization with Kibana. Increased visibility is vital for data analysis and helps identify errors to reduce mean time to recovery (MTTR).

Rounding Out The DevOps Toolchain for 2021

Programming isn’t only about shipping quality code, it’s about efficient execution. This drive to improve operations across the board is truly putting Ops into everything. With awesome open source DevOps tooling, it means more and more architects can adopt a DevOps approach within their deployment models.

DevOps as a practice, as well as its underlying technology, is always evolving. In 2021, a lot of effort is being placed on wrangling the effects of introducing a microservices development style. We’ve noticed the maturation of container orchestrators and service mesh, as a result. Blueprinting infrastructure-as-code and creating automated, repeatable configurations is another must-have for implementing automated build and release pipelines. Not to mention, many no-code and low-code platforms are opening up DevOps capabilities to non-programmers, albeit from a proprietary perspective.

It’s also helpful to note that some popular open source DevOps tools have been acquired — like Docker and Chef — blurring the lines between their business and their open source roots. When it comes to open source tools, it’s a good practice to adopt vendor-neutral tools with solid community support representing diverse stakeholders. This will help future-proof your project. Even though open source is “free,” users must ensure the benefits outweigh the time-to-onboard as well as the operational overhead of self-hosting.

DevOps tools are beneficial to automate software deployment. Here, we’ve attempted to list some of the most exciting tools to help automate that process. Of course, many new tools are emerging in the DevOps space, and we have hardly scratched the surface. Do you have a favorite DevOps tool that’s not mentioned above? Let us know in the comments below!

December 23, 2021
Rails 7 FTW | WFH is SOP | 586M Passwords Dumped

Welcome to The Long View—where we peruse the news of the week and strip it to the essentials. Let’s work out what really matters.

This week: Ruby on Rails 7.0 is go, working from home is still de rigueur, and HIBP gets far, far bigger.

DHH is Excited

First up this week: Ruby on Rails version 7.0 is now in production. It promises less bloat, better security and even more developer efficiency.

Analysis: Last chance for greatness

Talk to devs about Ruby on Rails and you’ll hear a wide range of opinion—all the way from “obsolete memory hog” to “productivity secret weapon.” But its important promise seems to be to encourage devs to use better design patterns.

Paul Krill: Ruby on Rails 7 rejiggers JavaScript support

Ruby on Rails 7, the latest version of the web application framework, is now available as a production release. … It gives developers a “no-Node” approach to front-end development while still providing access to NPM packages and modern JavaScript.
…
Most Rails applications will not require Node.js, given the new defaults. … Other improvements:
…
Encrypted attributes have been added to Active Record, enabling applications to offer at-work encryption in addition to traditional at-rest and in-transit. … With asynchronous query loading, two unrelated queries can be run concurrently. … The Spring application preloader is no longer on by default, as faster computers have made it unnecessary.

Let’s hear from Rails creator David Heinemeier Hansson—AKA DHH:

Seven is … the culmination of years of progress on five different fronts at once. … The part that really excites me … is how much closer it brings us to the ideal of The One Person Framework. A toolkit so powerful that it allows a single individual to create modern applications upon which [to] build a competitive business.
…
It’s been well over twenty years for me now, and a big release like this still makes me giddy. … What a glorious time to be working in web development.

What a whatnow? Here’s bb_matt:

I don’t pretend … it’s a “glorious time to be working in web development,” except for one important aspect: … Experienced programmers coming over from software development and starting to make amends [with] strong typing, robust CI/CD, a focus on TDD, a focus on domain modeling.
…
99% is still total **** though—but then again, so is most software.

Home Working Ain’t Gone Away

A month ago, my long-suffering editor permitted me to write, “WTH? We Wanna WFH.” And survey after survey says tech workers work better at home. After the last 21 months, there’s simply no denying it.

Analysis: Genie out of bottle—no going back

If you’re under the misapprehension that DevOps is somehow “special” and your workers will be glad to return—think again. Find a way to make it work, or work will stop—and your business will die.

Steven J. Vaughan-Nichols: How to lure employees back to the office? You can’t. Not now. Not ever.

Months have gone by, and the great resignation keeps rolling along. … People don’t want to catch COVID-19 [but] bosses still think they can force skilled workers to return to offices. … That’s not going to happen.
…
The problem? Many executives and owners haven’t gotten the clue yet. [But] any doubts you may have about people not doing a good job unless you’re looking over their shoulders should have vanished by now. … The monthly academic WFH research.com survey has found almost six out of 10 workers reported being more productive working from home. … In the Dice State of Remote Work report … 53% of technologists listed greater productivity as one of the main benefits.
…
It’s simple: It works better for them and for your company. … Anyone who’s been paying attention … must agree. You can either go along with the flow, or you can fight it and first lose your staffers and then your company

So what do DevOps managers think? One of them is kozikow:

100% remote is not great either. … When there was a long period of 100% remote during Covid, there were some issues. … We developed a policy that Mondays are required, Wednesday are encouraged, rest is work from home. … It was shared decision of the team.
…
I think that most people who are strongly against hybrid work just have to live in a miserable place like SF. … It’s unfortunate that so many software companies are in the Bay Area.

But you couldn’t pay some people enough to go back—people such as this Anonymous Coward:

No way in hell they’d ever be willing to pay me … to come back into an office. … No having to go out in ****ed up weather, no fighting traffic and … dangerous drivers, plus I get all that commuting time back.
…
And I don’t have to put up with Linda cooking fish in the microwave.

Have I Been Pwned? There’s a 38% Higher Chance

This week saw a huge bump in the size of Troy Hunt’s already-massive database of stolen credentials. In case you’ve not seen it, HaveIBeenPwned.com is an amazing free service that lets apps securely check a user’s password isn’t already used elsewhere on a compromised site.

Analysis: Build the API into your auth flow ASAP

The UK’s equivalent to the FBI has donated a huge trove of credentials—after deduping, it’s added more than 225 million new passwords to the database. Frankly, if you still allow your users to rely on any of the 847 million passwords in HIBP, it would be negligent.

Simon Sharwood: National Crime Agency finds 225 million previously unexposed passwords

Troy Hunt, of Have I Been Pwned (HIBP) fame, yesterday announced … the NCA shared 585,570,857 [creds] with HIBP, and Hunt said 225,665,425 … he hasn’t seen before in the 613 million credentials HIBP already stored. [HIBP] lets anyone conduct a secure search of stolen passwords to check if their credentials have been exposed.
…
The NCA’s statement to Hunt did not reveal the source of the password trove [but said they] “were able to identify a huge amount of … passwords in a compromised cloud storage facility. … It became clear that these credentials were an accumulation of breached datasets known and unknown.”
…
[The] release brings the total Pwned Passwords count to 847,223,402, a 38 percent increase.

How’s this relevant to DevOps? Troy Hunt explains:

In the last month, there were 1,260,000,000 occasions where a service somewhere checked a password against … HIBP’s Pwned Password API. … There are all sorts of amazing Pwned Passwords use cases out there. For example … checking their customers’ passwords on every registration, login or password change to see if it’s previously been breached.
…
The UK’s National Crime Agency has done some wonderful work over the years to combat cybercrime. … Today’s release is about turning on the firehose of new passwords and making them immediately available to everyone for free. Having this open to the community, owned by the community and supported by the FBI and NCA is an enormously pleasing result.

Time for a metaphorical thought experiment. LenKagetsu obliges:

You were told by a reliable source that an unknown person … posted a high-res image of … your house key … on the internet. Would you change the locks?

The Moral of the Story: Never switch horses in midstream

You have been reading The Long View by Richi Jennings. You can contact him at @RiCHi or tlv@richi.uk.

Image: David Emrich (via Unsplash)

December 21, 2021
Best of 2021 – Torvalds’ Bug Warning is a Lesson for Linux Users

As we close out 2021, we at devopsy.kinsta.cloud wanted to highlight the most popular articles of the year. Following is the third in our series of the Best of 2021.

Linux does, occasionally, raise security concerns. While many users see it as the most secure, robust and versatile operating system available — that’s this writer’s opinion, as well — security precautions still have to be taken.

A recent, widely publicized case illustrated this point; Linux creator himself, Linus Torvalds, warned against the use of the Linux 5.12 release. He described a “nasty bug,” and wrote that the situation is a “mess,” due to the use of swap files when adding Linux updates. This nasty bug, in fact, had the potential to destroy entire root directories.

Some of the main takeaways following this “mess” include: tread very carefully when installing early Linux releases, especially those that involve swapping files instead of partitions, and especially, despite Linux’s well-known security advantages, avoid becoming complacent, because Linux security is not always foolproof.

Hence, while the “state of Linux security today is quite good, and has evolved in a positive way with more visibility and security features built, like many operating systems, you must install, configure and manage it with security in mind; that is how cybercriminals take advantage, [via] the human touch,” said Joseph Carson, chief security scientist and advisory CISO at Thycotic, a provider of privileged access management (PAM) solutions.

A Patch for Nastiness

As Torvalds noted a few weeks ago, “most people don’t use a swap file, but a separate swap partition and the bug in question really only happens when you have a regular file system, and put a file on as a swap.”

“The bad news is that the reason we support swap files in the first place is that they do end up having some flexibility advantages, and so some people do use them for that reason. If so, do not use [release candidate] RC1,” Torvalds wrote. “Thus, the renaming of the tag.”

After issuing the warning, Torvalds released a patch that he says prevents the bug from destroying swap file systems. However, it may have already been too late for early adopters of release 5.12. Ubuntu, a leading Linux distro, can swap files by default.

“It is nasty bug if you are still using swap files,” Carson said. “If you do still use swap files, then you could be impacted, resulting in potential data loss or a corrupted system.”

DevOps teams – or anyone else running Linux and installing patches, whether on multi-servers or on individual workstations – still need, of course, to follow strict best practices. “Like any operating system, security depends entirely on how you use, configure or manage the operating system,” Carson said. “Each new Linux update tries to improve security; however, to get the value, you must enable and configure it correctly.”

Linux Goodness

The fact that Torvalds was so forthcoming about the bug, as well as the level of transparency that the Linux kernel offers, also demonstrates one of the many reasons Linux remains popular. Given that the Linux kernel, in one variety or another, is used “not only in about 50% of the internet servers of the world, but also in a substantial part of all our smartphones, it is good to see this level of transparency at ‘root level,” said Dirk Schrader, global vice president, security research at New Net Technologies (NNT), which providers cybersecurity and compliance software.

“The security of Linux is based on its transparency; the ability to review the code of a distribution,” says Schrader. “Quite often forgotten is that transparency also involves talking about the mistakes, the errors, those nasty bugs.”

Citing National Institute of Standards and Technology (NIST) vulnerability database statistics, Schrader described how, compared to the Windows family of desktop and server operating systems, for example, the Linux kernel shows better results for overall vulnerabilities. The number of vulnerabilities have also declined over the past four years, while Microsoft’s operating systems do not display the same trend, according to NIST’s national vulnerability database.

Since Linux’s famous kernel is open source and transparent, it is possible to extrapolate that there are a greater number of potential vulnerability watchdogs compared to those monitoring vulnerabilities in closed systems. Some may argue that Microsoft has been, at times, less successful at detecting vulnerabilities and issuing much-needed patches.

However, Linux users still must remain vigilant.

“Still, for any of the Linux distributions, anyone using the early release candidates — RC1 in particular — should make sure that their own development or build process is undergoing change control, so that no mishaps will transfer the nasty bug into a production environment,” said Schrader.

December 21, 2021