devopsy.kinsta.cloud

Category: DevOps and Open Technologies

Editorial Calendar for February 2016

GitLab Revamps Observability Strategy After Acquiring Opstrace

GitLab plans to make observability a more integrated element of its continuous integration/continuous delivery (CI/CD) platform after acquiring Opstrace.

Opstrace’s open source project exposes a horizontally scalable application programming interface (API) created by Cortex for Prometheus, an open source monitoring platform being advanced under the auspices of the Cloud Native Computing Foundation (CNCF). It then makes use of a Loki API to collect log data.

DevOps teams can also point an existing instance of Prometheus to Opstrace or any instance of Fluentd or Promtail software they are using to collect log data, with support for a wide variety of other APIs, such as the Datadog agent, planned. A command line interface (CLI) is provided to set an instance of Opstrace that creates a controller through which the collecting of log data is managed.

Kenny Johnston, senior director for product management for GitLab, said as part of an ongoing effort to make observability more accessible, GitLab plans to integrate Opstrace by default with GitLab Monitor for organizations that use either the self-managed version of GitLab or the software-as-a-service (SaaS) platform.

Opstrace currently collects logs and metrics and support for distributed traces is planned. The goal is to extend the overall scope of observability capabilities that GitLab already provides within its CI/CD platform, noted Johnston. The goal is to make it simpler for DevOps teams to employ observability capabilities and interrogate data to optimize workflows without acquiring a separate observability platform, he added.

GitLab is betting that, as Prometheus continues to gain traction, more DevOps teams will prefer to extend its reach into the realm of observability rather than having to manage a separate platform.

Although observability has always been a core tenet of DevOps best practices, achieving it has proven to be a challenge. At best, most organizations can continuously monitor IT environments based on predefined metrics that identify when a specific platform or application is performing within expectations. Observability combines metrics, logs and traces—a specialized form of logging—to instrument applications in a way that makes it simpler to troubleshot issues without having to rely solely on a limited set of pre-defined metrics that have been created to monitor a specific process or function.

The rate at which DevOps teams will embrace observability will naturally vary. In 2022, the biggest obstacle might not be the technology as much as it is simply understanding what queries can help DevOps teams better understand the root cause of an IT issue before there is a major disruption. In the long term, it’s expected that machine learning algorithms will leverage the data collected by observability platforms to automatically identify issues that might lead to a disruption long before it actually occurs.

One way or another, the ability to discern events even within the most complex IT environment is steadily improving. Whether that’s accomplished as an extension of monitoring or by embracing true observability may not be all that relevant—as long as the ultimate goal is achieved.

January 10, 2022
Intel Expands Scope of OneAPI Toolkit Initiative

Intel has made available a 2022 edition of its oneAPI toolkit through which it makes processor extensions accessible to DevOps teams.

James Reinders, chief evangelist for Intel software products, said the goal is to make it simpler for DevOps teams to expose extensions to any company’s processors in the runtimes developers use.

At its core, the oneAPI toolkit is a collection of APIs that Intel created for x86, graphics processing units (GPUs), application-specific integrated chips (ASICs), field programmable gate arrays (FPGAs) and other classes of processors.

Reinders noted that developers have never had as much ready access to so many different processor classes before. The challenge organizations face, however, is making it simpler for developers to take advantage of those capabilities in a way that doesn’t require them to write low-level code, he added.

The oneAPI 2022 toolkit adds additional capabilities including the world’s first unified compiler implementing C++, SYCL and Fortran, data-parallel Python for CPUs and GPUs, advanced accelerator performance modeling and tuning and performance acceleration for artificial intelligence (AI) and ray tracing visualization workloads. Collectively, Intel said it has made 900 new and enhanced features available via the oneAPI toolkit in the last year, including deeper Microsoft Visual Studio Code integration, support for Microsoft Visual Studio 2022 and Microsoft WSL2 for Linux development on Windows.

As IT continues to evolve it will become more routine for developers to invoke heterogeneous classes of processors within the same application, said Reinders. Those capabilities will drive an explosion of software development innovation through the rest of the decade and beyond, he added.

Intel is especially focused on making it easier for developers to invoke inference engines that take advantage of processor extensions to run AI workloads, Reinders said. Going forward, most applications will take advantage of machine learning algorithms that run inference engines optimized by Intel via extensions to its processors. As those workloads are constructed, integrating AI models within DevOps workflows will become the next major challenge, noted Reinders.

It’s too early to say how simplified access to heterogeneous processors, also known as XPUs, might change the application development landscape. The one thing that is clear is IT organizations now have more options than ever. The issue now is finding a way to invoke all those capabilities in a way that doesn’t overwhelm developers. DevOps teams will play a key role in deciding which processors and associated extensions to make available to their development teams.

In the meantime, the battle for processor dominance will only intensify in the months ahead. DevOps teams will need to decide how comfortable they are using processors manufactured by a single company like Intel or whether they want to employ a mix of processors provided by multiple vendors. Regardless of what they decide, it appears APIs will play a critical role in making it easier to benefit from advances in processor technologies.

January 7, 2022
Theranos’s Holmes VC Reverb | 5G C-Band Marches On | Mozilla Jumps Shark

Welcome to The Long View—where we peruse the news of the week and strip it to the essentials. Let’s work out what really matters.

This week: The conviction of Theranos CEO Elizabeth Holmes leads to greater scrutiny of startups, higher-frequency 5G NR is creeping towards reality, and Mozilla gets criticized for promoting climate-harming scams.

1. Holmes Case Causes Valley Rethink?

First up this week: Elizabeth Holmes has been found guilty of defrauding Theranos investors. But how will the affair change Silicon Valley’s startup-and-funding culture?

Analysis: Don’t fake it—just make it.

This could be a seminal moment, causing investors to look more closely at startup founders and their outlandish promises. On the other hand, key Theranos investors—e.g., Murdoch, Cox, DeVoss—hardly fit the mold of Silicon Valley VCs.

David Streitfeld: The Epic Rise and Fall

F. Scott Fitzgerald’s creation … Jay Gatsby … was a bootlegger [and] sold fake bonds. Ms. Holmes chose Silicon Valley, the last and greatest of all human dreams. [She] was a natural salesperson, as good at bending reality as Steve Jobs himself.
…
She and her deputy and boyfriend, Ramesh Balwani … thought they could brazen it out in the Silicon Valley tradition until they had something that actually worked. [But] the verdict signaled the end of an era. In Silicon Valley, where the line between talk and achievement is often vague, there is finally a limit. … It is an epic rise and fall that will be chewed over in the coffee shops and juice bars of Palo Alto, Calif., until the tech industry departs [for] off-world colonies.
…
So where does this conviction leave … her investors and former fans? Ripe for the next huckster that comes along, probably. Some Silicon Valley promises are so sweet we just can’t get enough of them. Immortality. Crypto. Flying cars. Mars. Digital harmony. Wealth beyond compare.

Think different, thinks DeanonymizedCoward:

[There’s a] fine line between bombastic marketing spam, “fake it ’til you make it” pitches, and outright fraud. … The first is repugnant and sometimes unethical, but generally legal.
…
[But] If your product requires bull**** to get funded—even if it ultimately works—you’re a fraud. … Regulatory reforms … and a sort of corporate death penalty … can fix a lot of this by making it less financially practical to engage in ****ty behavior.

Who else is to blame? Ask DS999:

I am sure some of her investors didn’t really care whether it worked. They just wanted to get in early, ride the hype train up, and sell before actual results mattered.

2. FAA and 5G Carriers Kiss and Make Up

Higher-frequency 5G NR is creeping towards reality in the U.S., thanks to the FAA backing down. After the aviation regulator had vetoed C-band deployment—based on precious little evidence—it looks like as much as 280 MHz of extra spectrum is now ready to switch on.

Analysis: Nomadic DevOps gets more bandwidth

While band 77 is nowhere near the much-vaunted “FR2” extremely high frequency bands, this is a useful chunk of extra spectrum for mobile and nomadic apps. Next stop: ubiquitous Ka-band.

Jon Brodkin: FAA agrees not to seek any more 5G delays

The Federal Aviation Administration tentatively agreed not to seek any more 5G delays from AT&T and Verizon, potentially ending a battle over the aviation industry’s unproven claim that 5G … will interfere with airplane altimeters. [Their] C-Band spectrum licenses are for the frequencies from 3.7 GHz to 3.98 GHz.
…
The deal incorporates voluntary commitments that AT&T and Verizon previously made, including “C-Band radio exclusion zones” around airports for six months. [They] modeled the exclusion zones after those used in France, which are 910×2100 meters.
…
Assuming there are no further problems, AT&T and Verizon would be able to use their spectrum licenses without extra restrictions after July 5. … President Biden applauded the agreement … calling it “a significant step in the right direction.”

In summary? Here’s Dieter Bohn:

The agency in charge of airplanes is mad about wireless things. But the agency in charge of wireless things and the companies that make them lost a game of chicken.
…
GREAT JOB AMERICA. … I always said I wanted Internet Service Providers to be Dumb Pipes, but I never expected them to be this dumb.

Has anyone read the FAA’s detailed report? bburdge has:

I can see why the FAA is backing down here now, it only makes sense when your key report could be paraphrased as: “We got a couple grad students to run some basic model comparisons … using altimeter data from the aviation industry (with a proven track record of lying to us) and then went golfing for eight months before publishing … but didn’t bother … to actually test anything.”

3. Mozilla Forgets its Values

Firefox custodian, the Mozilla Foundation, has angered many by promoting a “grossly irresponsible” “Ponzi scheme” that harms the environment. What is this awful thing? I hear you ask. It’s cryptocurrency.

Analysis: Imaginary money is bad—is this the last straw for Mozilla?

Less than a year ago, Mozilla said it wants to be carbon-neutral, but it’s already reneged on that promise: Distributed proof-of-work systems require … well, work. And that work uses ridiculous amounts of energy per DeFi transaction. Electricity production is a zero-sum game, so cryptocurrencies and NFTs cause climate change by emitting ridiculous amounts of CO₂e.

Liam Proven: Mozilla founder blasts browser maker

A few days ago, Mozilla Foundation invited netizens on Twitter to send in cryptocurrency donations. … This move by the Firefox browser maker rapidly drew criticism.
…
As recently as September, Moz’s web-standards lead Tantek Çelik slammed cryptocurrencies as being unsustainable, stating: “[They] are harmful for sustainability. … Their energy requirements grow without any discernible upper bound, which is grossly irresponsible given the global environmental crisis.”
…
Mozilla has accepted crypto-coin donations for years, perhaps as part of its long-ongoing efforts to alienate its own users: It’s only the payment provider that changed.

Chief among critics was Jamie “jwz” Zawinski:

I founded Mozilla and I’m here to say **** you and **** this. Everyone involved in the project should be witheringly ashamed of this decision to partner with planet-incinerating Ponzi grifters.
…
Stop saying you were promised flying cars. Unless you were born in 1935, you weren’t promised flying cars, you were promised a cyberpunk corporate dystopia.

It’s a shark-jump moment thinks Jeremy Keith:

The most convincing argument I’ve heard yet for switching away from Firefox as my main browser comes from Mozilla. I wasn’t expecting that plot twist.

The Moral of the Story: What is safety to one is not always safety to another.

You have been reading The Long View by Richi Jennings. You can contact him at @RiCHi or tlv@richi.uk.

Image: Benjamin Hayward (via Unsplash)

January 6, 2022
Open Source Observability Comes of Age in 2022

Observability is growing in importance to meet envisioned business objectives. Despite the benefits to performance data, organizations have faced challenges in realizing the full potential of observability based on implementation, distributed software tools, and other technological hurdles. But with new innovation, the era of true observability is upon us.

Innovative changes

The open-source observability landscape, in particular, is going through a phase of innovation. Powered by distributed tracing “coming of age” with OpenTelemetry, and eBPF taking center stage in terms of restructuring as the next frontier of observability and monitoring on Linux, innovation is at an all-time high.

Kernel technology

Extended Berkeley Packet Filter (eBPF) has presented a fascinating technology native to the Linux kernel. Born with the intent of having a programmable way of dealing with networking, eBPF has evolved into a rather generic and performant facility to have safe, dynamic access to the Linux kernel, that enables, among many other things, to extract telemetry for observability and security purposes. eBPF is powerful and growing in popularity, and Microsoft has been working on a port to Windows.

Versatility

In terms of observability, eBPF proved to be remarkably versatile in 2021. For example, there are tools that use eBPF to perform distributed tracing, like the Pixie project that was open-sourced in mid-2021 by New Relic. But the most promising aspect of eBPF for 2022 is the fact that it is possible, with eBPF, to implement continuous profiling of applications in production. Examples of this are the aforementioned Pixie project and Parca. As proven by tools like Google’s Cloud Profiler and various commercial observability tools, having a low-overhead, always-on profiler in production is a powerful way to troubleshoot latency and memory issues in production environments, which are notoriously hard and time-consuming to reproduce in test labs.

Clear differences

While production profiling with eBPF is happening now, it is not yet a general capability for use with most applications. The way profiling works is surprisingly different across different runtimes and programming languages. While pretty much all of the languages behave similarly in terms of using the CPU and allocating memory (and even in such foundational computing matters, there are differences), the way process concurrency is notably different, pretty much across the board.

For example, Java is using threads inside the Java Virtual Machine that are mapped to threads in the Operating System (to be fair, there are also other options, like NIO and libraries like RxJava, Reactor, Vert.x, plus Project Loom will eventually ship). In Node.js, on the other hand, concurrency is mostly dealt with the Event Loop, and “real” threads at the Operating System are abstracted away from the developer. Differences like these matter for a profiler, because the data shown to the person troubleshooting an issue must be “translated” to the model for concurrency used by the programming language, and this requires explicit support in the profiler for specific runtimes.

Open source observability comes of age in 2022

Fortunately, 2022 is going to be the year that brings us full-featured, open-source, eBPF-based production profiling tools that understand the specifics of the many runtimes that are prevalent in today’s cloud-native applications, like Java, Node.js, Python and .NET, as well as compiled languages like Go and Rust. And that will be a game-changer for improving software and solving issues faster, for all DevOps and SRE and operators.

As we look out to the year ahead, revolutionary technology like eBPF means the rate of innovation happening at the operating system level is only picking up and setting the stage for the year of true observability.

January 6, 2022
DevSecOps Will Cross the Chasm in 2022

We’ve been talking about DevSecOps and shift-left security for years. Although this approach probably didn’t “cross the chasm” in 2021, we did see some very telling milestones. Cybersecurity VC funding surged to record heights with a focus on DevOps and cloud security and the need for DevSecOps became glaringly evident with the Log4j vulnerability. On a more personal note, our own open source security project Checkov has surpassed three million downloads.

Much of this momentum was inevitable. Those in infrastructure and reliability circles have already been adopting and touting the benefits of shifting left and automating as much as possible, so in many ways, it’s expected that security would follow suit. Undoubtedly, the pandemic has also accelerated the need for developers to work more autonomously without having security personnel and processes acting as barriers to productivity and velocity.

We’re confident that 2022 will see the silos between development and security teams continue to crumble as developer-led security practices become the norm for cloud-native organizations.

DevSecOps finally crossing the chasm doesn’t mean that every enterprise and traditional organization will shift security left and adopt security best practices. It will, however, give them competitive advantages over those who don’t—both in decreased security costs and increased developer productivity and, thus, time-to-market.

What else is in store for the future of DevSecOps?

Rise of the DevSecOps Job Title

Fewer than 5,000 people on LinkedIn currently have “DevSecOps” in their job title, yet there are over 20,000 current openings for DevSecOps roles. In 2022, expect to see more of those positions filled. This means security teams across industries will conduct fewer manual security audits, there will be a considerable influx of homegrown DevSecOps tooling and point solutions will likely begin to consolidate into single platforms.

Blurred Lines Between Application and Infrastructure Security

Until recently, application security was a very well-defined (albeit fractured) space focusing on securing the custom code and open source packages that make up applications. However, with the rapid adoption of cloud-native applications, the lines between application and infrastructure security are blurring. We expect to see this trend continue as more engineers take on more infrastructure-related projects, vendors start catering to use cases outside of their core competencies (through acquisitions and in-house development) and the role of DevSecOps continues to expand within organizations.

Infrastructure-as-Code: The Great Cloud Migration’s Next Chapter

We’ve been talking about the great migration to the cloud for years. At this point, many companies are ready to move on to the next chapter: Infrastructure-as-code (IaC). As a result of this trend, DevSecOps will become much more important, as security needs to be baked earlier in development phases or risk being left behind. In addition, security teams will need to become more well-versed in development technologies and practices to provide the proper guidance for the new way applications are built and deployed.

More Software Supply Chain Attacks

Hackers have been targeting retailers and security vendors for years, exploiting a minor weakness to gain access and move laterally into sensitive data. In the past year, however, software supply chain attacks stepped into the spotlight due to multiple supply chain attacks. We expect this trend to continue, which will, in turn, put more focus on securing supply chains. DevSecOps, which has focused mostly on tools and practices for securing the code and infrastructure, will expand to include the supply chain mechanism.

If 2021 was the year of hype for DevSecOps, we believe (and hope) that these best practices are embraced en masse across industries. The benefits of shift-left security are well documented: The number of high severity incidents is significantly reduced, the potential attack surface is minimized, compliance efforts are simplified and the time to remediation is lowered. Organizations also save money by catching misconfigurations and vulnerabilities earlier in the software development life cycle while at the same time gaining time back with tools, both open source and commercial, that are empowering developers to move fast and build applications that are more secure and reliable.

January 5, 2022
IoT Developers: Get Familiar With LoRaWAN for 2022

IoT Analytics forecast the global number of connected IoT devices would grow to 12.3 billion active endpoints in 2021. That number is expected to grow exponentially in the near future as more and more industries make use of long-range wireless capability. For example, president Biden’s infrastructure plan promised major investment in smart cities and smart municipalities across the United States. 2022 will also see IoT growth in the manufacturing and utility sectors as ‘brownfield industries’ look to update their existing machinery for Industry 4.0. Both of these factors will bring about a massive surge in demand for low power wide area network (LPWAN) solutions using the long range wide area network (LoRaWAN) standard next year.

LoRaWAN is the standard protocol for LoRa wireless technology. LoRa was built specifically for long range use cases (it’s in the name). This means that technology using LoRaWAN requires significantly fewer repeaters and gateways to cover a large area, like a city, than would be required using traditional technologies. Additionally, LoRa is robust in how it responds to interference, and LoRaWAN sensors use less energy than other sensors, making them more efficient. In fact, ABI Research forecast that, by 2026, LoRa will be the leading non-cellular LPWAN technology.

In the grand scheme of IoT, LPWAN technology is relatively new. LoRaWAN became an open standard just six years ago when the LoRa Alliance was formed in 2015. This means the main challenge for IoT developers in 2022 and beyond will be educating themselves on the technologies needed to put LoRa-enabled plans into action. There are two big reasons why you should begin that process now: The rising demand for smart cities and Industry 4.0.

Building Cities Smarter

When it comes to infrastructure, roads and bridges might come to mind first, but in reality, infrastructure is much more holistic. Utilities like electricity, gas and telecommunications should all be included in any infrastructure discussion. These industries have already come to rely on IoT technologies to collect data and function in the modern world.

The government IoT endpoint electronics and communications market is expected to reach $21.3 billion in spending in 2022, according to projections by Gartner. As governments continue to modernize and implement more wireless technology, I expect a good amount of this money to go toward the development of solutions using LoRaWAN to create flexible networks on a very large scale for smart cities.

In fact, this movement has already begun. In September, the city of San Jose, California, announced its partnership with a blockchain-based provider to extend the city’s broadband access. In exchange, the city is now hosting and expanding its peer-to-peer IoT network infrastructure.

Updating Brownfield Industries for Industry 4.0

2022 will also see IoT advancements in ‘brownfield’ industries. These are industries with legacy, stand-alone machines and processes, such as in the manufacturing and utilities sectors, that weren’t originally created with internet connectivity in mind. Players in brownfield industries already have large install bases of machines that have no way of reporting status or other pertinent data.

These industries have long been looking at developing IoT solutions to solve this issue. Most are prototyping now with the goal of getting data to flow between large, legacy manufacturing machines with the end goal of improving manufacturing processes. However, unlike other industries where modern updates are often needed to improve efficiency, these manufacturers aren’t going to replace equipment that already gets the job done and has done so for decades. Instead, they need to find an alternate way to gather data and insights from the equipment they already have in place.

For many, LoRa-enabled sensors will be the answer. Often, upgrading to new technology requires equipment be substantially changed. That isn’t the case with LoRaWAN solutions, which allow manufacturers to add LoRa-enabled sensors to existing machines. LoRa-enabled sensors are durable, don’t require much maintenance and have low battery footprints. They also allow manufacturers to keep their data on-premises and on their own internal networks, a common preference for these types of industries. These characteristics make LoRa-enabled sensors ideal for manufacturing purposes.

In the new year, we will continue to see more pilot deployments go into production to help these industries gather the data they need to improve their processes and, in return, their bottom lines.

LoRaWAN Resources for Developers

The intention of the Biden administration’s high-tech infrastructure plans in the United States and the benefits of LPWAN (and LoRaWAN in particular) for private, long-range networks are converging to present a lot of great opportunities for this technology this year. That’s why 2022 will be the year IoT developers need to brush up on their knowledge of LoRaWAN.

If you are interested in learning more about LoRa, check out the LoRa Alliance or LoRa Developer Portal communities to get started.

January 4, 2022
The Future of OpenTelemetry for Observability

The observability landscape is changing. Today’s digital transformation journey has highlighted the need for observability as organizations look to capture, analyze and correlate all data across every level of the application stack including logs, metrics, traces and events. With the rampant rise of data, collecting these signals is imperative to both business insights and system optimization.

Over the last several years, open source data collection has given software and engineering teams the freedom to decide which technologies they want to standardize on for data collection and which vendors they would like data sent to. While open source data collection brought a lot of freedom and advantages, it also created some new problems. The lack of a single, comprehensive framework for all the critical signals observability tracked created a lot of friction for developers. As we move into the new year, businesses are taking the opportunity to reflect on how they can better use their data and apply intelligence to help customers make decisions and sift through all the noise. Enter OpenTelemetry.

What is OpenTelemetry?

OpenTelemetry is a collection of tools, APIs and SDKs that instrument, generate, collect and export all machine data; it was born from the need to develop an industry-wide standard for software instrumentation. By leveraging a single cloud-native framework to complete observability, OpenTelemetry is becoming the gold standard for collecting machine data.

OpenTelemetry consists of three primary components: Specification, collector and instrumentation. The specification uses an API, SDK and the semantic conventions and protocol of OpenTelemetry to determine the cross-language requirements for all implementations. The collector removes the need for multiple data collection agents. It is a vendor-neutral method for receiving, processing, transforming and exporting data. The instrumentation libraries allow users to integrate their applications with OpenTelemetry using auto-instrumentation and support from multiple frameworks and languages. Together they provide a robust framework for companies to disrupt their existing observability practices and amplify them even further.

It’s All About the Customers

Most modern enterprises have complex distributed microservices consisting of tens, hundreds or thousands of components that are all working together to deliver on customer experience. If organizations can’t correlate between those components or the different signals easily, it will slow down a business’s ability to effectively monitor, diagnose and troubleshoot issues.

For years, organizations have leaned into observability by relying on multiple infrastructure vendors to gather, enrich and ship data to vendors. Still, many still do not maximize observability’s most pronounced ability: Correlating complex systems and signals.

As end users desire deep insight into their data, they also expect vendors to offer the best experience to identify and resolve issues within their systems with the help of observability. With true observability, organizations can give customers back their time to focus on what matters, which is managing the digital experiences for their customers and empowering them to meet their customers’ needs.

A Similar Road to Adoption

Much like Kubernetes has been essential for the growing adoption of software containers, the industry is already witnessing the same adoption approach around OpenTelemetry for observability. Kubernetes won the war for becoming the standard for container orchestration. Now, OpenTelemetry will bring the same standards needed for observability with comprehensive visibility into the health and performance of every layer of the stack.

OpenTelemetry has become widely adopted by the software development community, but it’s still at the beginning of its journey. The developer community made Kubernetes a success, and we will see the same with OpenTelemetry as customers demand better integration and less lock-in. For vendors involved around applications and software—from more traditional approaches like application performance management to software development pipelines and management tools and through to cloud-native services—supporting OpenTelemetry will be table stakes in the future.

With extensive portability, greater developer control and strong vendor and cloud provider support, OpenTelemetry has opened a new world to customers by helping to improve achievable results across the application infrastructure, from the software development process to areas like security.

How can businesses continue to move the needle on observability while balancing growing customer demands? Quite simply: If organizations aren’t thinking about OpenTelemetry now, they are losing competitive advantage by lacking the proper visibility to deliver on the service level objectives that customers expect in this day and age.

January 3, 2022
New Relic: Observability Tops 2022 DevOps Agenda

If 2021 was the year when IT organizations became aware of the need for observability, then the coming year should see observability aspirations realized.

A report published by observability platform provider New Relic postulates that there are no less than 11 factors converging in a way that will make observability an absolute requirement for organizations.

At the top of that list is a simple fact: Observability is now required to drive digital business transformation initiatives that typically span multiple processes. In fact, a global survey of 1,300 IT leaders, software engineers and developers conducted by New Relic found 90% of respondents view observability as critical to the business.

Second, as a result, observability is now starting to be applied in both pre- and post-production environments as organizations take a more holistic approach to managing the software development life cycle, the report noted.

Third, the report suggested that IT teams are looking to make better-informed decisions based more on facts than intuition. To do this, teams are using data collected from across the enterprise application environment. In fact, the fourth factor advancing observability in the enterprise is that the cost of instrumenting application environments is dropping, which the report notes is making it easier to collect relevant data.

The fifth and sixth factors reflect a desire to rationalize the number of monitoring tools that IT teams currently use and the fact that usage-based pricing has simultaneously made it less expensive to leverage observability platforms as needed.

The seventh factor is the overall shift toward open source software that enables observability platform providers to collectively leverage innovations faster. The eighth factor, closely linked to the open source trend, is the rise of more complex open source platforms such as Kubernetes that require much higher visibility to successfully deploy and manage.

The ninth trend identified in the report is increased adoption of DevOps best practices that encourage IT teams to view observability as a core tenet of any IT management strategy. The tenth factor cited by the report is the need to drive increased collaboration among IT professionals.

Finally, the eleventh factor driving further adoption of observability is the need for increased application reliability as more organizations appreciate their dependence on software to deliver essential services.

Buddy Brewer, a group vice president and general manager for New Relic, said now that observability platforms are becoming more accessible, the next big challenge will be teaching IT professionals what questions to ask as they troubleshoot both legacy monolithic applications and a wide range of emerging microservices-based applications. The results of those queries then need to be easily shared with both developers and internal IT operations teams that must work together to maximize application performance and availability, he added. The latter requirement is why New Relic decided to acquire CodeStream to make telemetry data available with integrated development environments (IDEs), noted Brewer.

There is, of course, no shortage of observability platforms for IT teams to adopt. In fact, the biggest challenge now is getting every application stakeholder to agree to standardize on an observability platform—no small feat, as many DevOps teams may already have a preference for one approach over another.

January 3, 2022
Best of 2021 – The Next Evolutions of DevOps

As we close out 2021, we at devopsy.kinsta.cloud wanted to highlight the most popular articles of the year. Following is the seventh in our series of the Best of 2021.

As society continues to recover from the global COVID-19 pandemic and some semblance of normal starts to return, technology continues to move forward. The pandemic certainly changed what normal means and people, processes and technologies all have adapted and evolved to meet those challenges. The narrative that the pandemic caused of your organization’s digital transformation is true for many practitioners. With 2022 just around the corner, DevOps is on the cusp of maturation of previously bleeding-edge paradigms and a focus on engineering efficiency. Here are five DevOps evolutions you should keep an eye on going forward.

Engineering Efficiency is Front and Center

Engineering efficiency is a particularly broad umbrella term. At the foundational level, engineering efficiency focuses on making someone (for example, an engineer) more productive. There are a multitude of disciplines that intersect, from organizational design to engineering/developer experience. Engineering efficiency is increasingly critical for organizations today. The pandemic had two major impacts on the technology sector in terms of resources and staffing. The first, at the start of the pandemic, was the unprecedented and unpredictable availability of resources; physical locations were being shut down because of the medical severity of the pandemic and the “getting hit by a bus” factor was playing out in real life. Second, during the recovery period, the Great Resignation began, and with it, a fight for resources that has become increasingly intense. Both have meant that the availability of resources, especially ramped resources, is scarce.

The Spotify model of tribes/guilds of engineering resources is a promising evolution; folks can frequently move around, reducing the toil that engineering resources face and allowing for engineering resources to not only ramp up more quickly but also increasing engagement and retention. The industry continues to march toward standardization in areas that were typically bespoke, such as the release process, and I think we’ll see that continue to evolve, as well.

Git is Ubiquitous

Leveraging a source code management (SCM) solution used to be reserved for software engineers. But as the proliferation of “something-as-code” has touched multiple levels of the technology stack from networking, storage and, ultimately, a development pipeline, the march of operations-focused engineers adopting many of the traits of software engineers continues. Having iteration in your infrastructure stack with more ephemeral/disposable infrastructure is becoming the norm. Saving and packaging the multitude of configurations for the “something-as-code” source control is a natural area of evolution.

Building off of source control are the package managers—your Docker registries, Helm chart repositories, etc.—for more deployable artifacts; that said, getting those artifacts to production is a process. Because Git is becoming ubiquitous across the technology landscape, this leads to the continued adoption of GitOps. Since the seminal piece was written by Weaveworks in 2017 defining the GitOps paradigm, leveraging GitOps is seen as a viable paradigm for many organizations five years later, especially when starting with greenfield initiatives.

Kubernetes is No Longer Bleeding Edge

In 2022, Kubernetes will celebrate the eighth birthday of its first commits on GitHub. Taking a jog down technology memory lane, eight years before Kubernetes (in 2006), VMware was still a private company and it would be a few years before vSphere was even released (remember how it felt to run a workload on a virtual machine in 2014?). The Kubernetes ecosystem still moves quickly, but placing workloads on Kubernetes is not a novel concept anymore. As application infrastructure and architecture have adopted the Kubernetes way (in other words, being idempotent and ephemeral), there is maturity around running a suitable workload on Kubernetes.

Kubernetes by design is highly pluggable. If you do not like the opinion or implementation of something inside Kubernetes, you can replace that opinion. Don’t like how Kubernetes handles ingress traffic? Then choose from one of the many ingress controllers available today. That is just one example of dozens of pluggable areas. Because of this, Kubernetes is viewed as a dynamic, non-static resource. Using Kubernetes takes trial and error and getting your cluster to be robust, performant and reliable is an ongoing journey that requires ongoing iteration.

Authors Can Be the Enforcers

There is no question that in a modern software stack, we are inundated with data. The metrics soup and the continued science of and exploration of observability (metrics, traces, logs) provide us with a lot to work with. Making decisions, even automated ones, based on that data continues to be critical. With the rise of site reliability engineering (SRE) practices, reacting or anticipating spikes in usage, failures and even security-related items are par for the course for modern teams.

These decisions can be authored into policies that specific platforms understand, such as autoscaling rules for a cloud vendor or something like Open Policy Agent in the Kubernetes ecosystem. With so many items shifting left toward the development team, finding the right resource, skillset or institutional knowledge to author these rules can involve the input of several teams. Because of the rise of “something-as-code,” regardless of whether the author of these rules is a software engineer, DevOps engineer, platform engineer or anyone in between, that author has the ability to enforce.

Shifting Left … But Complexity Shifts Left, Too

The old adage is that complexity is like an abacus: You can shift complexity around, but it never really goes away. With the movement to shift responsibility left to development teams, this also means that associated complexity is shifting to the development teams. Modern platform engineering teams provide the infrastructure (compliant Kubernetes clusters) to teams and any workload that is run on those clusters is up to the development team that owns it. Typically, development teams then focus on features and functionality. Managing lots of non-functional requirements—and even core infrastructure requirements such as networking—can be a burden; think about how your organization would handle a service mesh.

If you are a DevOps or platform engineer, making your internal customers—your development teams—successful is a great goal to work toward. Crucial to this is disseminating expertise. This can be in the form of automation and education. A common practice with the DevSecOps movement is to have some sort of scanning step as part of the build or deployment process, disseminating the internals as far as how the scan is performed, what happens if something is found, etc. Gaining internal adoption is a journey and having a good developer experience built around clarity and stability are important.

Evolution in 2022

Even with all of the challenges, bumps and learnings that the last year has given us, technology continues to evolve and advance and even lower the barriers of entry to continually become more inclusive. Focusing on improving engineering efficiency and reducing toil will allow for more participation and lower the barriers to entry.

December 23, 2021
Rails 7 FTW | WFH is SOP | 586M Passwords Dumped

Welcome to The Long View—where we peruse the news of the week and strip it to the essentials. Let’s work out what really matters.

This week: Ruby on Rails 7.0 is go, working from home is still de rigueur, and HIBP gets far, far bigger.

DHH is Excited

First up this week: Ruby on Rails version 7.0 is now in production. It promises less bloat, better security and even more developer efficiency.

Analysis: Last chance for greatness

Talk to devs about Ruby on Rails and you’ll hear a wide range of opinion—all the way from “obsolete memory hog” to “productivity secret weapon.” But its important promise seems to be to encourage devs to use better design patterns.

Paul Krill: Ruby on Rails 7 rejiggers JavaScript support

Ruby on Rails 7, the latest version of the web application framework, is now available as a production release. … It gives developers a “no-Node” approach to front-end development while still providing access to NPM packages and modern JavaScript.
…
Most Rails applications will not require Node.js, given the new defaults. … Other improvements:
…
Encrypted attributes have been added to Active Record, enabling applications to offer at-work encryption in addition to traditional at-rest and in-transit. … With asynchronous query loading, two unrelated queries can be run concurrently. … The Spring application preloader is no longer on by default, as faster computers have made it unnecessary.

Let’s hear from Rails creator David Heinemeier Hansson—AKA DHH:

Seven is … the culmination of years of progress on five different fronts at once. … The part that really excites me … is how much closer it brings us to the ideal of The One Person Framework. A toolkit so powerful that it allows a single individual to create modern applications upon which [to] build a competitive business.
…
It’s been well over twenty years for me now, and a big release like this still makes me giddy. … What a glorious time to be working in web development.

What a whatnow? Here’s bb_matt:

I don’t pretend … it’s a “glorious time to be working in web development,” except for one important aspect: … Experienced programmers coming over from software development and starting to make amends [with] strong typing, robust CI/CD, a focus on TDD, a focus on domain modeling.
…
99% is still total **** though—but then again, so is most software.

Home Working Ain’t Gone Away

A month ago, my long-suffering editor permitted me to write, “WTH? We Wanna WFH.” And survey after survey says tech workers work better at home. After the last 21 months, there’s simply no denying it.

Analysis: Genie out of bottle—no going back

If you’re under the misapprehension that DevOps is somehow “special” and your workers will be glad to return—think again. Find a way to make it work, or work will stop—and your business will die.

Steven J. Vaughan-Nichols: How to lure employees back to the office? You can’t. Not now. Not ever.

Months have gone by, and the great resignation keeps rolling along. … People don’t want to catch COVID-19 [but] bosses still think they can force skilled workers to return to offices. … That’s not going to happen.
…
The problem? Many executives and owners haven’t gotten the clue yet. [But] any doubts you may have about people not doing a good job unless you’re looking over their shoulders should have vanished by now. … The monthly academic WFH research.com survey has found almost six out of 10 workers reported being more productive working from home. … In the Dice State of Remote Work report … 53% of technologists listed greater productivity as one of the main benefits.
…
It’s simple: It works better for them and for your company. … Anyone who’s been paying attention … must agree. You can either go along with the flow, or you can fight it and first lose your staffers and then your company

So what do DevOps managers think? One of them is kozikow:

100% remote is not great either. … When there was a long period of 100% remote during Covid, there were some issues. … We developed a policy that Mondays are required, Wednesday are encouraged, rest is work from home. … It was shared decision of the team.
…
I think that most people who are strongly against hybrid work just have to live in a miserable place like SF. … It’s unfortunate that so many software companies are in the Bay Area.

But you couldn’t pay some people enough to go back—people such as this Anonymous Coward:

No way in hell they’d ever be willing to pay me … to come back into an office. … No having to go out in ****ed up weather, no fighting traffic and … dangerous drivers, plus I get all that commuting time back.
…
And I don’t have to put up with Linda cooking fish in the microwave.

Have I Been Pwned? There’s a 38% Higher Chance

This week saw a huge bump in the size of Troy Hunt’s already-massive database of stolen credentials. In case you’ve not seen it, HaveIBeenPwned.com is an amazing free service that lets apps securely check a user’s password isn’t already used elsewhere on a compromised site.

Analysis: Build the API into your auth flow ASAP

The UK’s equivalent to the FBI has donated a huge trove of credentials—after deduping, it’s added more than 225 million new passwords to the database. Frankly, if you still allow your users to rely on any of the 847 million passwords in HIBP, it would be negligent.

Simon Sharwood: National Crime Agency finds 225 million previously unexposed passwords

Troy Hunt, of Have I Been Pwned (HIBP) fame, yesterday announced … the NCA shared 585,570,857 [creds] with HIBP, and Hunt said 225,665,425 … he hasn’t seen before in the 613 million credentials HIBP already stored. [HIBP] lets anyone conduct a secure search of stolen passwords to check if their credentials have been exposed.
…
The NCA’s statement to Hunt did not reveal the source of the password trove [but said they] “were able to identify a huge amount of … passwords in a compromised cloud storage facility. … It became clear that these credentials were an accumulation of breached datasets known and unknown.”
…
[The] release brings the total Pwned Passwords count to 847,223,402, a 38 percent increase.

How’s this relevant to DevOps? Troy Hunt explains:

In the last month, there were 1,260,000,000 occasions where a service somewhere checked a password against … HIBP’s Pwned Password API. … There are all sorts of amazing Pwned Passwords use cases out there. For example … checking their customers’ passwords on every registration, login or password change to see if it’s previously been breached.
…
The UK’s National Crime Agency has done some wonderful work over the years to combat cybercrime. … Today’s release is about turning on the firehose of new passwords and making them immediately available to everyone for free. Having this open to the community, owned by the community and supported by the FBI and NCA is an enormously pleasing result.

Time for a metaphorical thought experiment. LenKagetsu obliges:

You were told by a reliable source that an unknown person … posted a high-res image of … your house key … on the internet. Would you change the locks?

The Moral of the Story: Never switch horses in midstream

You have been reading The Long View by Richi Jennings. You can contact him at @RiCHi or tlv@richi.uk.

Image: David Emrich (via Unsplash)

December 21, 2021