devopsy.kinsta.cloud

Tag: outage

  • Microsoft Outage Outrage: Was it BGP or DNS?

    Microsoft Outage Outrage: Was it BGP or DNS?

    Welcome to The Long View—where we peruse the news of the week and strip it to the essentials. Let’s work out what really matters.

    This week: All of Microsoft’s cloud services go down, everywhere except China. Redmond’s IaaS, PaaS and SaaS—including GitHub—were dead for several hours, and are still running unreliably—despite Microsoft saying it’s fixed the problem. (more…)

  • 5 Ways to Prevent an Outage

    5 Ways to Prevent an Outage

    In today’s always-on, ever-connected world, we all expect 100% availability.

    What gets in the way of this? The devil is in the details. Over time, everything breaks:  Disks, nodes, containers, networks, DNS servers and configuration issues can all lead to major outages.

    Amazon’s 13 minutes of downtime in August 2021 translated to almost $5 million in lost revenue and Amazon Web Services’ three outages in December of last year cost millions more. And it was widely reported that the October 2021 Facebook (including related products: Messenger, Instagram and WhatsApp) outage resulted in an estimated $100 million in lost revenue due to a network configuration error.

    There have been many outages of varying scope and duration—some attributed to technology breaking, others traced back to people making mistakes. In fact, a 2017 Amazon outage was attributed to “human error”–or more specifically, one employee’s typo. The employee intended to take a small number of servers offline to fix a billing system issue but the incorrect command led to several servers being affected and a major outage.

    So, what can your company do to reduce the risk, duration and impact of a potential outage? For one, companies can’t pretend that if they only try harder, they can prevent humans from making mistakes. Do you have dozens of people manually keying in hundreds of commands every day? If so, a mistake is inevitable. Companies should instead investigate how and why one small blunder in a command line could do widespread damage. Guardrails and redundancies should be in place to protect against and minimize these types of incidents.  

    As the enterprise continues its digital transformation toward a multi-cloud, hybrid and distributed world, here are five guidelines to consider—guidelines that can help you preempt and prevent your next big outage.

    1.  Always Consider Your Blast Radius

    Blast radius measures how one error, typo or malfunction can impact other parts of a system. 

    What is the chain of components or services that is impacted by one error? 

    What are the hidden dependencies? 

    How many customers would be affected and for how long?

    More is connected than you think; discover how it’s all related in advance to eliminate surprises.

    If too many systems in a network are mutually dependent and interconnected, the result is deadlock. In other words, when one breaks, the other also fails. We have seen this in past Facebook outages. A single erroneous command sparked a domino effect that shut down the backbone connecting all of Facebook’s data centers globally.

    How can you prevent this scenario? Plan for the worst possible scenario. When it comes to mission-critical systems, take an intentional approach to situation failover and backup. What if A, B and C all happen simultaneously? Also, think about your cost versus reliability trade-offs. Maybe you can’t afford two full sets of servers but can afford a full set of archived code. 

    2.   Smooth Out Rough Edges

    Consider that core infrastructure and code get used (and tested) a lot more than edge code/infrastructure. Basically, the core is hardened while edge infrastructure is more vulnerable. You should be more proactive at the edge around identifying vulnerabilities and doing explicit testing to counter this risk.

    The 2014 Google disk erase/satellite outage is a cautionary tale involving the edge. In Google’s case, the edge networks offered a key advantage: A better connection for the customer due to lower latency since data is accessed from a source closer to the user as opposed to the core network. Google had a CDN/edge network for TCP connection termination, caching for various apps, DNS regulation, etc. Teams needed to routinely upgrade a satellite rack and run decommission automation—and this automation failed part of the way through. Error message. They retried the decommission process again to debug the failure. The manual re-execution of the decommission automation meant thousands of satellite machines were decommissioned and, within minutes, the disks of all satellite machines globally were erased. The result: Customers experienced an increase in latency. Behind the scenes, it required two days of all-hands-on-deck to reinstall these machines. 

    3.   Think Escalator, Not Elevator

    A good system has both defense and depth. Considering that it takes three or four things to (really) go wrong, think about failure cascades. Bluntly put, limit dependencies. Measure the longest dependency failure path and mean-time-to-recovery (MTTR). When you do this, think escalator not elevator. What do I mean? Well, when an elevator fails, you are left with a shaft; an escalator that fails still works as a staircase.

    In 2016, severe weather in Sydney, Australia resulted in an AWS EC2/EBS outage. First, the utility provider suffered a loss of power at a regional substation. This failure resulted in a total loss of utility power to multiple AWS facilities. In one of the facilities, AWS power redundancy didn’t work as designed and power was lost to a significant number of instances in that availability zone. To make matters worse, the instances that lost power lost access to both their primary and secondary power as several power delivery lineups failed to transfer load to their generators. Also, a latent bug in AWS instance management software resulted in a slower-than-expected recovery of the remaining instances.

    The lesson? This correlated power delivery lineup failure illustrated that failures are not independent. You must design for the largest correlated failure in combination with the background noise of ongoing independent failure. Recovery code needs to be scale-tested against the largest expected failure. 

    4. Stay Ahead of Digital Certificate Rotations

    Don’t underestimate the importance of digital certificates—and their expiration dates. A few years ago, LinkedIn allowed one of its SSL certificates to expire and it knocked out LinkedIn sites in the U.S., UK and Canada. In 2017, Equifax missed a breach for 76 days because of an expired certificate. The average annual cost of unplanned outages due to certificate expirations is $11.1 million.

    For example, you could have a load balancer sitting in front of several servers. The load balancer could have an up-to-date certificate while the servers behind it are expiring. It is important to check for expiring servers in multiple ways. Ideally, you check the boxes in two ways: Ping the HTTP endpoints and also have scripts that check the certificate files themselves.

    Expired certificates (TLS for example) can lead to encryption and authentication risks as well as lost internet connectivity and larger outages involving websites and applications. To avoid this, you continuously check, renew and deploy certificates—and this can be achieved via automation. ​​Think about automatically rotating expiring certificates.  

    5.   Tackle Toil to Empower Your Teams

    What’s the biggest headache for site reliability engineers (SREs) and those who manage production operations? Toil. 

    We can define toil as manual, tedious tasks that engineers perform within production environments. Basically, toil wastes time and slows down operations. As a result, most SREs and engineering teams strive to minimize toil within their workflows.

    In production operations, you should work to limit toil. Make sure your team is regularly investing time in proactive work; the reactive work shouldn’t take most of their time. Toil reduction gives developers, DevOps engineers and SREs more time to focus on projects that create net-new value for the business (new application features). Google stated that it has an “advertised goal” of keeping toil “below 50% of each SRE’s time.” Basically, no engineer should spend more than half of their time on manual, repetitive work. But the ultimate goal should be to get toil as close to zero as possible. Automating incident remediation can help SREs spend less time on toil and less time on-call.

    Also, when you implement fixes—especially people and process fixes—ask yourself if this is a systemic fix or one that is relying on best intentions. Look for tools to safely empower your team (with guardrails). As you work to identify major vulnerabilities and dependencies (as discussed above), you should automate away the most common issues while planning for the “corner cases” (or the 1% of 1%) that may lead to major outages (a.k.a. the worst-case scenarios).

    As you consider these guidelines, make sure you focus on putting actual mechanisms in place—not one-off, “hopeful” directives. Do not rely on people to remember best practices from a training session—these are not mechanisms. Instead, design processes that can be systematically implemented. When you do a post-mortem, look for concrete programs you can put into place that will prevent the same mistake from happening again. Automation can help you achieve this level of confidence, management and quality control—and it may just keep you one step ahead of a potential future outage.

  • Cloudflare Outage Outrage | Yet More FAA 5G Stupidity

    Cloudflare Outage Outrage | Yet More FAA 5G Stupidity

    Welcome to The Long View—where we peruse the news of the week and strip it to the essentials. Let’s work out what really matters.

    This week: Cloudflare suffers another huge outage while the FAA and FCC still disagree over 5G/NR near airports. (more…)

  • What SREs Can Learn From the Atlassian Outage of 2022

    What SREs Can Learn From the Atlassian Outage of 2022

    What happens when the tools and services you depend on to drive site reliability engineering turns out to be susceptible to reliability failures of their own? That’s the question teams at about 400 businesses presumably asked themselves in the wake of a major outage in Atlassian Cloud. The incident offers a number of insights for SREs about reliability risks within reliability management software itself—as well as how to work through complex outages efficiently and transparently.

    What Caused the Atlassian Cloud Outage?

    The outage, which began on April 4 and finally resolved April 18, affected about 400 Atlassian Cloud customer accounts. Atlassian Cloud is a hosted suite of popular Atlassian products including Jira and OpsGenie. The outage meant that affected customers could no longer access these tools or the data they managed in them.

    According to Atlassian, the problem was triggered by a faulty application migration process. Engineers wrote a script to deactivate an obsolete version of an application. However, due to what Atlassian called a “communication gap” between teams, the script was written in such a way that it deactivated all Atlassian Cloud products, not just the obsolete application.

    To make matters worse, the script was apparently configured to delete data permanently, rather than mark it for deletion, which was the intention. As a result, data in affected accounts was removed permanently from production environments.

    The Good, the Bad and the Ugly

    The Atlassian Cloud outage may not be the very worst type of incident imaginable—failures like Facebook’s 2021 outage were arguably worse because they affected more people and because service restoration was complicated by physical access issues—but it was still pretty bad. Production data was permanently deleted and hundreds of enterprise customers experienced total service disruptions that have lasted several days and counting.

    Given the seriousness of the incident, it’s tempting to point fingers at Atlassian engineers for letting an incident like this happen in the first place. They seem to have written a script with some serious issues, then presumably deployed it without testing it first—which is exactly the opposite of what you might call an SRE best practice.

    On the other hand, Atlassian deserves credit for responding to the incident efficiently and transparently. Although the company was silent at first, it ultimately shared details about what happened and why even though those details were a bit embarrassing to its engineers.

    Crucially, Atlassian also had backups and failover environments in place, which it used to speed the recovery process. The major reason why the outage lasted so long, the company said, is that restoring data from backups to production requires integrating backup data for individual customers into storage that is shared by multiple customers, a tedious process that Atlassian apparently can’t perform automatically (or doesn’t want to, presumably because it would be too risky to automate).

    Unfortunately for impacted customers, it does not appear that any fallback tools or services were made available while they waited for Atlassian to restore operations. We imagine this poses more than minor issues for teams that rely on tools like Jira to manage projects and OpsGenie to handle incidents. Perhaps those teams stood up alternative tools in the meantime – or perhaps they crossed their fingers, hoping their project and reliability management tools would come back online ASAP. 

    Takeaways for SREs From the Atlassian Outage

    For SREs, then, the key takeaways from this incident would seem to be:

    • Always perform dry runs of migration processes in testing environments before putting them into production. Presumably, if Atlassian engineers had tested their application migration script first, they would have noticed its flaws before it took out live customer environments.
    • Back up, back up, back up—and make sure you have failover environments where you can rebuild failed services based on backups. While this outage is bad, it would be 100 times worse if Atlassian couldn’t restore service based on backups and data had been lost permanently.
    • Ideally, each customer’s data should be stored separately. As we noted above, the fact that Atlassian used shared storage seems to have been a factor in delaying recovery. That said, it’s hard to fault Atlassian too much on this point; it’s not always practical to isolate data for each user due to the cost and administrative complexity of doing so.
    • SRE teams would do well to think about how they’ll respond if their reliability management software itself goes offline. For example, it might be worth extracting and backing up data from your reliability management tools so you can still access it if your tool provider experiences an incident like this.
    • Over-communicate with your customers often and early. In this case, there was initial radio silence that left customers in the dark. Most of this chatter eventually took place in public forums.

    Conclusion

    The Atlassian Cloud outage is notable both for its length and for the fact that, ironically, it took out software that teams use to help prevent these types of issues from happening at their own businesses.

    The good news is that Atlassian had the necessary resources in place to restore service as quickly as possible. A shared data storage architecture led to slow recovery, which is unfortunate; again, it’s hard to blame Atlassian too much for not setting up dedicated storage for each customer.

  • Apple Outage Outrage | Linux Random Redo | Okta Hacked (or Not)

    Apple Outage Outrage | Linux Random Redo | Okta Hacked (or Not)

    Welcome to The Long View—where we peruse the news of the week and strip it to the essentials. Let’s work out what really matters.

    This week: Why Apple services were down, Linux gets a huge RNG overhaul, and we wonder if Okta was hacked again. (more…)

  • AWS Outage Exposes Weaknesses of DevOps Resilience

    AWS Outage Exposes Weaknesses of DevOps Resilience

    The December 7, 2021 Amazon Web Services (AWS) outage severely disrupted services from a wide range of businesses for more than five hours and highlighted just how reliant businesses have become on internet-delivered services. The outage mostly impacted web services in the eastern U.S., yet the implications are universal: It’s a reminder that many businesses blindly ignored the old axiom about putting all your eggs in one basket and instead are relying on a provider with a single point of failure.

    Services ranging from airline booking systems to streaming video to e-commerce were disrupted during the outage, causing millions of dollars in lost revenue and countless hours in productivity. One of the more interesting aspects of the outage is the impact it had on services from collaboration vendors such as Slack, Trello, Asana and Smartsheet—tools that many development and DevOps teams have come to rely on.

    Furthermore, core AWS services, such as the company’s Elastic Compute and DynamoDB cloud tools were also impacted, disrupting many third-party services and severely hampering business processes that use those services. While the obvious victims of the outage are well known, like Amazon’s own e-commerce operation, there is a troubling undercurrent: The disruption to DevOps frameworks and those using them.

    AWS has been rather tight-lipped about the root cause of the outage thus far; however, there are still many lessons to be learned for the DevOps community and questions that must be asked such as “Can the DevOps process survive during IaaS/SaaS disruptions?” and “Can multi-cloud failover solutions be baked into the applications DevOps builds?”

    There are no simple answers to those questions, but the outage highlights the need to understand the underlying architecture and framework of a deployed DevOps system. Take, for example, how many developers in the DevOps community have embraced SaaS tools to accelerate the development process and to feed CI/CD pipelines. SaaS applications such as code scanners, pipeline orchestration and even IDEs have become common in the world of DevOps. But has anyone bothered to ask what happens if a single one of those tools fails?

    What’s more, the reliance on SaaS tools in the development process has led to the creation of potential liabilities in the applications created by DevOps developers. DevOps applications have come to rely on APIs, are often driven by microservices and are frequently deployed into containers that run on SaaS. If any of those elements become non-functional, numerous applications could fail, ultimately putting the onus on developers to explain why they are creating applications with a single point of failure.

    Moving forward, the DevOps community needs to take a serious look at the components of their frameworks and determine if there are any single points of failure—including their IaaS/SaaS providers. While it may be impossible to remediate every single one, there is a lesson to be learned about how fragile the development process can become if no one bothers to build an inventory of the tools used and take into account how a failure of any one of those tools could impact workflow. There are numerous examples of how an external failure of a single component impacted the functionality of an application, while the discovery process of the root cause has taken days or even weeks. This can be mostly attributed to not only a lack of knowledge, but a lack of visibility into the components used.

    Those lessons can be extended to the development process itself, where the best practice of rooting out single points of failure can be extended to the applications themselves. Leveraging that intelligence starts with understanding the concept of a software bill of materials (SBoM), a piece of supporting documentation that is becoming increasingly important to the purveyors of applications. A properly-defined SBoM reveals all of the components (libraries, APIs, etc.) that are baked into an application and can be used as a map to define where weaknesses may lie.

    For the DevOps community, the recent AWS outage has become a clarion call to look inward and discover how the applications they are building may be part of the problem. With continuity and resiliency becoming major topics in the IT and business realm, it’s about time that DevOps practitioners start to look at how they can support both of those business-critical needs. The days of finger-pointing to shift blame must come to an end, and if businesses that rely on software want to grow, someone needs to take responsibility for providing answers when outages occur and learn from those outages to create applications that are more resilient.

  • AWS Outage Outrage | Rusty Linux | ARM Latest

    AWS Outage Outrage | Rusty Linux | ARM Latest

    Welcome to The Long View—where we peruse the news of the week and strip it to the essentials. Let’s work out what really matters.

    This week: Amazon Web Services falls on its face, Linux’s move to Rust takes the next step, and the FTC stabs another fatal wound in the horrible Arm/Nvidia deal.

    AWS us-east-1 Breaks the Internet

    First up this week: Amazon Web Services’ outage on Tuesday sent scads of DevOps teams scrambling. But what was the cause?

    Analysis: DNS DDoS?

    While we still don’t officially know, there is some chatter to speculate on. Could it be that AWS’s DNS infrastructure is vulnerable to DNS reflection attacks?

    Hello. It’s Dave Lee: From angry Adele fans to broken robot vacuums

    It started early on Tuesday morning. Robot vacuums ceased sucking, Wi-Fi cameras stopped watching and eager Tinder daters were left unable to “swipe right.”
    …
    Within Amazon itself … the unthinkable occurred: grounded delivery drivers were unable to load packages and deliver to customers’ doorsteps, just as the peak Christmas season begins to step up. … Drivers at multiple facilities across the country were sent home … some dreading whatever workload may await.
    …
    An outage at Amazon Web Services … had rippled through the online economy, crippling services used by millions of people. … An “impairment of several network devices” in … US-EAST-1 was the “root cause” of the disruption, Amazon said.


    I was wondering what DevOps’ers learned? Enter Max Rozen:

    AWS took approximately an hour to update their status page to indicate anything was happening. … While our infrastructure is hosted across 20 AWS regions, a single point of failure was hosted in us-east-1.
    …
    What we learned: … I learned the saying, “Friends don’t let friends deploy in us-east-1.” As a result, we’ll be moving some things over to us-east-2 to start with. Secondly, we’ll be adding redundancy and the ability to failover to another AWS region.


    Getting inside the business, here are Eugene Kim and Katherine Long, to go over everything:

    Even inside AWS … information on the outage remained sketchy [and] rumors spread among staff. One AWS employee speculated that the outage was caused by an “orchestrated DNS attack,” while another employee downplayed those concerns.
    …
    According to a screenshot of an internal AWS communique … ”firewalls are being overwhelmed.” … A separate internal note said … the network issue “specifically impacted” Amazon’s internal DNS servers.

    Linux Officially Gets Rust Code

    The Rust language has taken the next step toward being part of Linux kernel development. Some “drivers and other non-core kernel programs” written in Rust can now be PR’ed. But there’s still no plan to go further.

    Analysis: Still needs lubrication

    Rust is much better than C for writing memory-safe code, and coding provable implementations. However, bootstrapping the toolchain is still stupidly hard.

    Steven J. Vaughan-Nichols: Linux is getting more Rust in it by the day

    It wasn’t that long ago that the very idea that another language besides C would be used in the Linux kernel would have been laughed at. [But] this has been coming for several years now. … Why? Because it’s much safer than C, especially at handling memory errors.
    …
    The next “patch series [adds] support for Rust as a second language to the Linux kernel.” … While he’s encouraging a slow but steady approach to introducing Rust into Linux, [Linus Torvalds] has also said that using Rust interfaces for drivers and other non-core kernel programs makes sense. [But] no one’s going to rewrite the kernel’s 25 million lines of C in Rust.
    …
    The Rust code proposed for the kernel now relies on the stable Rust compiler rather than the beta compilers. … As Linux kernel and lead Rust on Linux, developer Miguel Ojeda, put it, “By upgrading the compiler, we have been able to take off the list a few unstable features we were using.”


    How is it “safer than C”? ssokolow shares this fascinating insight:

    Rust tries to encourage that through good API design which allows it to be proven at compile time that the invariants will be upheld, thus avoiding the need to choose between safety and performance. My favourite example … is “the typestate pattern” which lets you teach the Rust compiler to prove correct traversal of any state machine.
    …
    A more mundane example would be the iterator APIs: No need for bounds checking if the iterator looks up the length once at the beginning and the APIs prevent you from wandering out of each entry it hands you. … Rust programs tend to run faster than their C and C++ counterparts because the developers feel comfortable introducing and maintaining more aggressive optimizations.


    “And then a miracle occurs,” quips @dave_universetf:

    I’m intrigued [but] I can’t help but feel sorry for the people doing bootstrapping work. C is hard enough to bootstrap, but Rust is a whole new level of pain.
    …
    Rust is literally not source bootstrappable. [The compiler] that works right now is Rust 1.39. So then you have to compile 18 more Rust compilers to get up to the current version.

    Nvidia’s ARM Woes Continue

    As I’ve said before, this deal really shouldn’t go through. Ownership of the ARM ISA and the core designs would be a huge strategic stick to beat Nvidia’s competition with. ARM chips are an increasing fixture in the datacenter—especially those that value “performance per Watt.”

    Analysis: Stick a fork in it

    Nvidia’s attempt to buy Arm from SoftBank is suffering death by a thousand cuts. And the latest wound comes from the FTC’s Higonokami knife.

    Paul Thurrott: FTC Sues to Block Sale of Arm to Nvidia

    The U.S. Federal Trade Commission … sued to block Nvidia’s $40 billion acquisition of … Arm, citing competitive concerns. … Nvidia announced its intention to purchase Arm from Softbank for about $40 billion, over one year ago, in September 2020. But the deal fell under immediate criticism from the major industry players that rely on Arm’s … designs, and the UK government announced … it would “intervene” for national security reasons.
    …
    Now, it may not happen at all. “We will continue to work to demonstrate that this transaction will benefit the industry and promote competition,” Nvidia said.


    Explain like I’m 5? Here’s swillden:

    ARM has a non-trivial and increasing share in the data center. … ARM is much, much bigger than phones … Arm is an important competitor to Intel and AMD. … Arm licensing IP to Qualcom, nVidia, Mediatek, Unisoc, Samsung, Apple, Google, etc., has been great for many areas of technology.
    …
    Arm acting as a common … designer for a bunch of different chip vendors has been extremely good for … progress in CPU/SoC design. … It would be really unfortunate if a lot of the industry were to shift back to x86, eliminating all of the competition on hardware features that we get from the many ARM chipmakers.


    But what of SoftBank? Ian King runs the numbers:

    Wall Street analysts are all but calling the deal dead. … According to Citibank analysts, the chances of the Nvidia purchase closing are now just 5%.
    …
    The company faces the prospect of losing a jackpot of roughly $74 billion. … Other suitors are unlikely to match such an offer. … So SoftBank’s best alternative may well be an initial public offering for the Arm business—but that is likely to value the company at … $19.8 billion—only about 27% the value of the … Nvidia deal.

    The Moral of the Story: It’s no secret that the both of us are running out of time


    You have been reading The Long View by Richi Jennings. You can contact him at @RiCHi or tlv@richi.uk. (It’s so typical of me to talk about myself—I’m sorry.)

    Image: Ləman Şirinzadə (via Unsplash)

  • Nvidia/ARM Wavering | Google Outage Outrage | Backblaze IPO on Fire

    Nvidia/ARM Wavering | Google Outage Outrage | Backblaze IPO on Fire

    Welcome to The Long View—where we peruse the news of the week and strip it to the essentials. Let’s work out what really matters.

    Nvidia Can’t be Allowed to Buy Arm

    First up this week: Nvidia’s faltering attempt to buy Arm: Regulators around the world are alarmed by the idea. And now the UK—the legal home of the company—has taken the next step, which could nix the deal.

    Analysis: RISCy home for ARM

    This matters because ownership of the ARM ISA and the core designs would be a huge strategic stick to beat Nvidia’s competition with. Arm doesn’t make chips—it licenses technology to Nvidia and many of its competitors—AMD, Apple, Broadcom, Samsung and Qualcomm, to name but five. So this deal really shouldn’t go through, regardless of Nvidia’s fluffy PR reassurances.

    ARM chips aren’t just found in 99% of smartphones, but are an increasing fixture in the datacenter—especially where “performance per Watt” is important. They’re even found in exascale iron, including the Riken Fugaku, the world’s fastest supercomputer.

    Helen Thomas: Nvidia faces tough Arm wrestle

    It is one thing for a regulator to give your deal a thorough going-over. It is another for it to imply publicly that your assurances [aren’t] credible. … Nvidia got both from the UK’s Competition and Markets Authority. … The CMA’s full phase 1 report doesn’t make for easy reading.
    …
    The transaction, which was announced in September 2020, is bogged down in regulatory reviews around the world: … Arm’s “neutral” business model has always made it a difficult acquisition. … Arm licenses its designs to other chipmakers, providing scope to restrict rivals’ access.
    …
    Arm was a political talking point in 2016, when its sale to SoftBank … was waved through with embarrassing ease. There is now a much more muscular political take on UK national security concerns with a particular focus on tech.


    But if it’s owned by a Japanese company, why is the UK getting involved? Glad you asked—Aighearach answers:

    What did they buy? Shares of a UK company. … Then it is a UK company! … Buying shares is buying ownership, but it doesn’t move the company to another jurisdiction.
    …
    This blathering about “sold to a Japanese firm” is getting old.


    Nvidia claims, “The transaction will … boost competition and innovation.” Falmari finds that hilarious:

    Did Nvidia’s spokesperson previously work as a stand-up comedian? Because as jokes go, “boost competition” is a whopper. [Do you think] that’s what a company like Nvidia wants—to boost the competition?

    Google Cloud Outage has Ripple Effect

    Google’s load balancers all went offline for two hours on Tuesday. Aside from the knock-on Google Cloud failures, DevOps tools such as BigCommerce, Datadog, Deno, Discord, Egnyte, Evernote, Linear, Netlify and Toggl were reported to be affected. But perhaps the worst impact was felt by the poor, unfortunate users of Pokémon GO.

    Analysis: Paralysis

    In theory, putting all your eggs in one basket is okay—so long as that basket can reliably stop your eggs smashing on the pavement. But this isn’t the first time we’ve seen Google Cloud issues such as this; I can’t imagine it’s the last.

    Steven J. Vaughan-Nichols: Google glitch triggers major internet outage

    The result? Many of us have seen 404 errors when trying to get to some of our favorite web pages. … There appears to be more major sites having problems than not.
    …
    The specific problem seems to be with the GCP load balancers. These, working with Google’s Cloud Delivery Network (CDN) [are] designed to stop website failures and Distributed Denial of Service (DDoS) attacks by putting your website behind a single anycast IP.


    Are you feeling a touch of déjà vu? jeffbee, too:

    This is a sort of repetitive outage for Google. They’ve wiped out the GSLB configs before.

    A year ago there was also that big outage where they blanked out the whole Gmail delivery config and started rejecting all mail. … Config safety is not their strong suit.


    The #3 IaaS provider won’t get to #2 if it keeps this up. Here’s u/totopapa:

    I’m trying to convince my company to switch from AWS to GCP but an outage like this one is definitely a showstopper.

    Backblaze IPO Burns Brightly

    Backblaze is now NAS:BLZE. In its first week, the stock jumped 60%.

    Backblaze is well known for its tiny-margin consumer backup product and its open-source “storage pod” hardware—but is less well known for its DevOps-focused “B2” cloud storage service.

    Analysis: Can it Avoid Death-by-Packrats?

    Backblaze is trying to grow this newer DevOps business, without losing its classic consumer chops. But many of its former competitors have essentially dumped their “unlimited” offerings—it’s the old tragedy-of-the-commons problem.

    Jordan Novet: Backblaze [goes] public on small valuation and minimal funding

    In an era of blockbuster market debuts preceded by astronomical private financing rounds, Backblaze … a cloud service for backing up data … is a throwback to a time when sub-billion-dollar companies routinely went public to raise capital and build their profile … to win the trust of customers.
    …
    Backblaze shares climbed 24% in their debut on Thursday and another 12% on Friday, lifting the price to $22.31 and giving the company a market cap of around $650 million. … By the standards of its peers, Backblaze is a micro-cap.
    …
    One of Backblaze’s competitors is Amazon, the world’s fourth most valuable U.S. company. … Backblaze currently stores two exabytes of data — that’s 2 million terabytes — and has a half-million paying customers.


    Donovan Jones: Backblaze Targets $100 Million IPO

    The firm provides cloud-based storage services to small businesses and individuals worldwide. BLZE has grown revenue and gross profit impressively through the 2020 pandemic period and the IPO appears fairly valued, so is worth a close look.
    …
    The firm’s financials show strong revenue and gross profit growth. … The firm’s dollar-based net revenue retention rate for 2020 was 114%—a solid result. … The market opportunity for providing cloud-based storage services to small businesses and individuals is large and expected to grow at a material rate. … As for valuation, compared to partial competitor Dropbox, management is seeking the same Enterprise Value/Revenue multiple. [However] the combined revenue growth rate and EBITDA percentage rate … was 3% as of June 30, 2021, so the firm needs improvement in this regard.


    But there’s a long way to go, opines John Kozubik (aka rsync):

    Of the ~500,000 customers, 70,000 are B2 customers and 430,000 are “backup” customers—which are flat-rate “unlimited” users. … Roughly 85% of the user-base are paying $7/mo for “unlimited backup.”
    …
    These flat-rate users are very fickle and price-sensitive. … Roughly 85% of the user-base are paying $7/mo for “unlimited backup.” … It’s a fools errand to pursue this market and I assume the pivot to B2 was to build something more sustainable with a much higher price and usage ceiling.

    The Moral of the Story: Success is a journey, not a destination


    You have been reading The Long View by Richi Jennings. You can contact him at @RiCHi or tlv@richi.uk.

    Image: Kristin Wilson (via Unsplash)

  • When IT Disaster Strikes, Part 3: Conducting a Blameless Post-Mortem

    When IT Disaster Strikes, Part 3: Conducting a Blameless Post-Mortem

    In the first and second parts of this three-part series, we looked at how organizations can effectively resolve incidents and the role of each person involved in the resolution process. In this final piece, we’ll look at what organizations must do post-incident to better understand what went wrong and learn from their response.

    At the time of occurrence, an outage is every IT practitioner’s worst nightmare. However, after the dust settles, each incident presents an opportunity to learn as a team, make an impact on your processes and improve your product. This means that even after you’ve found a resolution, your work isn’t quite done.

    Enter the Incident Post-Mortem

    Once you’ve successfully fixed a major issue and the chaos of the firefight is winding down, it’s time to conduct an incident post-mortem with your team to investigate the problem and figure out what went wrong. Without a post-mortem, you fail to recognize what you’re doing right, where you could improve and, most importantly, how to avoid making the same mistakes in the future.

    Here are my three tried and true steps for a successful post-mortem process:

    Designate a Post-Mortem Owner

    To kick off the post-mortem, the incident commander should assign an owner at the very end of an incident. This owner is responsible for scheduling the post-mortem meeting, which should take place within five business days of an incident. Owners are also responsible for populating the post-mortem document, investigating the incident, pulling in other people to assist and facilitating the post-mortem meeting. In cases where a public blog post is required, the owner will also be responsible for creating the content and putting it through necessary internal review cycles.

    Populate the Post-Mortem Document and Conduct Analysis

    Once an owner has been designated, the team should get to work updating the post-mortem document with available information, including the timeline of status changes and key actions taken by incident responders. Every participant in the incident response should be included in this page, so be sure to go back through all communication and add relevant parties. After you’ve updated the page with the full timeline of actions, you will need to build out details for each event. For each item in the timeline, identify a metric or a third-party page where the data came from. This could be anything from a link to a Datadog graph or Splunk search to a relevant Tweet, as long as it shows the data point you’re trying to illustrate.

    With the timeline in place, you can begin analyzing the incident. Capture all available data about what happened, what caused the issue, how many customers were affected, what impact they experienced, etc., to determine the underlying cause of the incident. This information should be detailed, as it will dictate follow-up action tickets your team will use to prevent recurrence of the incident and improve the incident response process. It also will be examined during the post-mortem meeting, so the more context you can include in your analysis, the better.

    In the case of severe incidents where external communication is required, this is also the time when the post-mortem owner and team should develop customer communications. Pro tip: Avoid using the word “outage” if possible in this communication. Many people see the word outage and assume that your entire system was down, when in reality that was likely not the case. Be precise here, as it is important for both internal learning and sharing within the larger IT community.

    Here is a general template for what to include on the post-mortem page:

    • Overview
    • What Happened
    • Root Cause (or Contributing Factors)
    • Resolution
    • Impact
    • Responders Involved
    • Timeline
    • How’d We Do?
    • Action Items
    • Communication (Internal and External)
    • Hold a Post-Mortem Debrief Meeting

    The final step in the process, the incident post-mortem meeting, should include the incident commander, service owners, key engineers/responders and the customer liaison (for severe incidents). Use your post-mortem document as an agenda to help things run smoothly. The discussion should center around what happened, what the team could have done better and any followup actions that need to be taken. Any disagreements about the facts, analysis or recommended actions should be cleared up during this time, and everyone should walk away from the meeting with a better sense of problems impacting your service’s reliability and having learned more about the service involved.

    The No. 1 rule of running an incident post-mortem is to keep it blameless. The goal of the debriefing process is not to point fingers, but to learn what happened and how you can improve as a team.

    Don’t make the mistake of neglecting the post-mortem process after a major incident. A well-designed, blameless post-mortem empowers your team to continuously learn and offers a way to iteratively improve your infrastructure and incident response process. Not only does this help your team, but it also helps create amazing software and services.

    — Eric Sigler