Category: CloudOps

Welcome to The Long View—where we peruse the news of the week and strip it to the essentials. Let’s work out what really matters.

This week: Amazon Web Services falls on its face, Linux’s move to Rust takes the next step, and the FTC stabs another fatal wound in the horrible Arm/Nvidia deal.

AWS us-east-1 Breaks the Internet

First up this week: Amazon Web Services’ outage on Tuesday sent scads of DevOps teams scrambling. But what was the cause?

Analysis: DNS DDoS?

While we still don’t officially know, there is some chatter to speculate on. Could it be that AWS’s DNS infrastructure is vulnerable to DNS reflection attacks?

Hello. It’s Dave Lee: From angry Adele fans to broken robot vacuums

It started early on Tuesday morning. Robot vacuums ceased sucking, Wi-Fi cameras stopped watching and eager Tinder daters were left unable to “swipe right.”
…
Within Amazon itself … the unthinkable occurred: grounded delivery drivers were unable to load packages and deliver to customers’ doorsteps, just as the peak Christmas season begins to step up. … Drivers at multiple facilities across the country were sent home … some dreading whatever workload may await.
…
An outage at Amazon Web Services … had rippled through the online economy, crippling services used by millions of people. … An “impairment of several network devices” in … US-EAST-1 was the “root cause” of the disruption, Amazon said.

I was wondering what DevOps’ers learned? Enter Max Rozen:

AWS took approximately an hour to update their status page to indicate anything was happening. … While our infrastructure is hosted across 20 AWS regions, a single point of failure was hosted in us-east-1.
…
What we learned: … I learned the saying, “Friends don’t let friends deploy in us-east-1.” As a result, we’ll be moving some things over to us-east-2 to start with. Secondly, we’ll be adding redundancy and the ability to failover to another AWS region.

Getting inside the business, here are Eugene Kim and Katherine Long, to go over everything:

Even inside AWS … information on the outage remained sketchy [and] rumors spread among staff. One AWS employee speculated that the outage was caused by an “orchestrated DNS attack,” while another employee downplayed those concerns.
…
According to a screenshot of an internal AWS communique … ”firewalls are being overwhelmed.” … A separate internal note said … the network issue “specifically impacted” Amazon’s internal DNS servers.

Linux Officially Gets Rust Code

The Rust language has taken the next step toward being part of Linux kernel development. Some “drivers and other non-core kernel programs” written in Rust can now be PR’ed. But there’s still no plan to go further.

Analysis: Still needs lubrication

Rust is much better than C for writing memory-safe code, and coding provable implementations. However, bootstrapping the toolchain is still stupidly hard.

Steven J. Vaughan-Nichols: Linux is getting more Rust in it by the day

It wasn’t that long ago that the very idea that another language besides C would be used in the Linux kernel would have been laughed at. [But] this has been coming for several years now. … Why? Because it’s much safer than C, especially at handling memory errors.
…
The next “patch series [adds] support for Rust as a second language to the Linux kernel.” … While he’s encouraging a slow but steady approach to introducing Rust into Linux, [Linus Torvalds] has also said that using Rust interfaces for drivers and other non-core kernel programs makes sense. [But] no one’s going to rewrite the kernel’s 25 million lines of C in Rust.
…
The Rust code proposed for the kernel now relies on the stable Rust compiler rather than the beta compilers. … As Linux kernel and lead Rust on Linux, developer Miguel Ojeda, put it, “By upgrading the compiler, we have been able to take off the list a few unstable features we were using.”

How is it “safer than C”? ssokolow shares this fascinating insight:

Rust tries to encourage that through good API design which allows it to be proven at compile time that the invariants will be upheld, thus avoiding the need to choose between safety and performance. My favourite example … is “the typestate pattern” which lets you teach the Rust compiler to prove correct traversal of any state machine.
…
A more mundane example would be the iterator APIs: No need for bounds checking if the iterator looks up the length once at the beginning and the APIs prevent you from wandering out of each entry it hands you. … Rust programs tend to run faster than their C and C++ counterparts because the developers feel comfortable introducing and maintaining more aggressive optimizations.

“And then a miracle occurs,” quips @dave_universetf:

I’m intrigued [but] I can’t help but feel sorry for the people doing bootstrapping work. C is hard enough to bootstrap, but Rust is a whole new level of pain.
…
Rust is literally not source bootstrappable. [The compiler] that works right now is Rust 1.39. So then you have to compile 18 more Rust compilers to get up to the current version.

Nvidia’s ARM Woes Continue

As I’ve said before, this deal really shouldn’t go through. Ownership of the ARM ISA and the core designs would be a huge strategic stick to beat Nvidia’s competition with. ARM chips are an increasing fixture in the datacenter—especially those that value “performance per Watt.”

Analysis: Stick a fork in it

Nvidia’s attempt to buy Arm from SoftBank is suffering death by a thousand cuts. And the latest wound comes from the FTC’s Higonokami knife.

Paul Thurrott: FTC Sues to Block Sale of Arm to Nvidia

The U.S. Federal Trade Commission … sued to block Nvidia’s $40 billion acquisition of … Arm, citing competitive concerns. … Nvidia announced its intention to purchase Arm from Softbank for about $40 billion, over one year ago, in September 2020. But the deal fell under immediate criticism from the major industry players that rely on Arm’s … designs, and the UK government announced … it would “intervene” for national security reasons.
…
Now, it may not happen at all. “We will continue to work to demonstrate that this transaction will benefit the industry and promote competition,” Nvidia said.

Explain like I’m 5? Here’s swillden:

ARM has a non-trivial and increasing share in the data center. … ARM is much, much bigger than phones … Arm is an important competitor to Intel and AMD. … Arm licensing IP to Qualcom, nVidia, Mediatek, Unisoc, Samsung, Apple, Google, etc., has been great for many areas of technology.
…
Arm acting as a common … designer for a bunch of different chip vendors has been extremely good for … progress in CPU/SoC design. … It would be really unfortunate if a lot of the industry were to shift back to x86, eliminating all of the competition on hardware features that we get from the many ARM chipmakers.

But what of SoftBank? Ian King runs the numbers:

Wall Street analysts are all but calling the deal dead. … According to Citibank analysts, the chances of the Nvidia purchase closing are now just 5%.
…
The company faces the prospect of losing a jackpot of roughly $74 billion. … Other suitors are unlikely to match such an offer. … So SoftBank’s best alternative may well be an initial public offering for the Arm business—but that is likely to value the company at … $19.8 billion—only about 27% the value of the … Nvidia deal.

The Moral of the Story: It’s no secret that the both of us are running out of time

You have been reading The Long View by Richi Jennings. You can contact him at @RiCHi or tlv@richi.uk. (It’s so typical of me to talk about myself—I’m sorry.)

Image: Ləman Şirinzadə (via Unsplash)