devopsy.kinsta.cloud

Tag: ML infrastructure

  • Google FLoC is Dead | Meta AI Supercomputer Lives | ARM Deal is Dead

    Google FLoC is Dead | Meta AI Supercomputer Lives | ARM Deal is Dead

    Welcome to The Long View—where we peruse the news of the week and strip it to the essentials. Let’s work out what really matters.

    This week: Google’s FLoC proposal is dead, Meta/Facebook is buying RSC—a huge AI supercomputer, and Arm “will IPO” instead of selling to Nvidia.

    1. Forget ‘Federated Learning of Cohorts’—Long Live ‘Topics’

    First up this week: Google bows to the inevitable. Third-party cookies will soon go away, because people are fed up with being tracked. But Google’s FLoC proposal wasn’t the answer.

    Analysis: FLoC is a flop

    Google doesn’t want to invade your privacy. But Google does want to sell well-targeted ads. The key difference between FLoC and Topics is that FLoC buckets were automatically generated—and hence opaque, risking overlap with “protected categories”—e.g., race or gender. But the topic classifications in the Topics API are curated and have user-visible, descriptive names.

    Frederic Lardinois: Google kills off FLoC, replaces it with Topics

    Google’s controversial project for replacing cookies for interest-based advertising … is dead. In its place, Google today announced a new proposal. … Your browser will learn about your interests as you move around the web.
    …
    When you hit upon a site that supports the Topics API for ad purposes, the browser will share three topics you are interested in. … The site can then share this with its advertising partners to decide which ads to show you. … Users will be able to review and remove topics from their lists [or] turn off the entire Topics API.
    …
    It’ll remain to be seen if other browser vendors will be interested in adding the Topics API. [But] they all quickly turned a cold shoulder to FLoC. … The plan is to start trialing the Topics API at the end of this quarter.


    A minor tweak and rename? “Yes and no,” equivocates mananaysiempre:

    Yes, it still amounts to building user-tracking functionality into user agents. No, it’s not the same approach.
    …
    FLoC was doing unsupervised clustering of users, whereas Topics allocates users into predetermined clusters. This makes it potentially more transparent and also seems designed to work around the “protected categories” objection.


    But Google has a long way to go to convince garett_spencley:

    I’m glad FLoC is being ditched, but as an end-user I’m still not asking for this Topics nonsense. It doesn’t solve any problem that I have. It solves problems that the advertising industry has.
    …
    I’ll choose to use browsers that don’t support this.

    2. Machine Learning at Metaverse Scale is Expensive

    Zuckerberg’s new toy is RSC: A $400 million Nvidia-based AI supercomputer. And that’s just Phase One.

    Analysis: 3.8×1018 integer operations per second is … a lot

    If you rely on Open Compute Project systems for your DevOps, know that it’s no longer the bee’s knees over at Hacker Way. Meta/Facebook is investing eye-watering sums into a custom configuration of off-the-shelf hardware.

    Timothy Prickett Morgan: Meta Buys, Rather Than Builds And Opens, Its Massive AI Supercomputer

    If you thought it took a lot of compute and storage to build Facebook’s social network, you ain’t seen nothing yet. … To build the Metaverse … is going to take an absolutely enormous amount of supercomputing power.
    …
    But what is a surprise, given Facebook’s more than decade long focus on … opening up its hardware designs through the Open Compute Project, is that the … Research Super Computer [RSC] system is built from … commercially available … servers, storage, and switches. [It] probably comes down to … working a deal with some vendors who want the Facebook publicity and are willing to work the price and the parts availability.
    …
    [So Meta bought] 760 DGX A100 systems from Nvidia—which each … have a pair of AMD “Rome” 64-core Epyc 7742 processors … 1 TB of main memory … eight A100 GPU accelerators [and] eight 200 Gb/sec HDR InfiniBand controllers—all-flash storage from Pure Storage, and … caching servers from Penguin Computing. … At list price this machine would cost around $400 million … Phase one of the RSC machine is rated at 59 petaFLOPS … (FP64), and 118.6 petaFLOPS FP64 using the Tensor Core matrix engines. … With INT8 processing on the Tensor Cores, which is needed for AI inference, the phase one machine is rated at 3.79 exaOPS.


    How much??? It doesn’t seem like news to kleiba:

    I used to work at a university where my professor had been in automatic speech recognition for a long time, but basically gave up on that line of research about 10 years ago because he figured that universities simply cannot compete budget wise with the big industry players. I suppose the same will soon be true for most ML-related areas of research sooner or later.
    …
    Already, a substantial amount of research innovation in [Natural Language Processing] and [Computer Vision] has been coming from big companies in recent years. Of course there is a discussion to be had about what that means for society at large.


    But 3.79 exoOPS? That’s the way the wind is blowing, according to sungazer:

    Give it 15 or so years—this level of computation will be on your desktop. If desktops still exist.
    …
    Or desks.

    3. Arm Bid: No Legs

    Speaking of Nvidia, as I’ve said before, the Arm merger really shouldn’t go through. And it’s looking less and less likely. ARM chips are an increasing fixture in the data center—especially those that value “performance per Watt.”

    Analysis: ARM M&A DOA—IPO PDQ

    A veritable hornets’ nest of Bloomberg journalists have put their names to an anonymously-sourced report. The story argues that the proposed Nvidia buyout of Arm from SoftBank Group is dead. Next stop: IPO.

    Ian King, Giles Turner, Peter Elstrom, Dina Bass, David McLaughlin, Ruth David and Dinesh Nair: Nvidia Quietly Prepares to Abandon … Arm Bid

    According to people familiar with the matter … Nvidia … doesn’t expect the transaction to close. … SoftBank, meanwhile, is stepping up preparations for an Arm initial public offering.
    …
    If Nvidia manages to get the deal over the line, it would be a massive coup. … But it will be an uphill fight. … The world’s biggest tech companies rely on Arm technology, and they fear they could lose unfettered access under Nvidia.
    …
    Within SoftBank, there are factions. [And] the ordeal has created divisions within Nvidia. … “We remain hopeful that the transaction will be approved,” a SoftBank spokesperson said in an emailed statement.


    Post-Brexit Britain would no doubt like to keep Arm independent. Here’s simonh:

    Arm has I think ⅔ of its employees in the UK—mainly Cambridge—and they’re expanding the campus there. The Semiconductor Physics Group at the Cavendish Laboratory at the University is at the cutting edge globally with 18 PhD projects currently.
    …
    Guess where scads of Arm’s top scientists are recruited from? Arm and the University are joined at the hip. That company’s not going anywhere.


    More broadly, it’s good news, thinks hatchet:

    Hopefully this means NVIDIA will focus on RISC-V instead. This will be good for everyone.

    The Moral of the Story: Zeal should not outrun discretion.


    You have been reading The Long View by Richi Jennings. You can contact him at @RiCHi or tlv@richi.uk.

    Image: Reed Geiger (via Unsplash)

  • Five Challenges of Machine Learning DevOps

    Five Challenges of Machine Learning DevOps

    As organizations add machine learning (ML) to their workflows, it’s tempting to try to squeeze model creation and deployment into the existing software development lifecycle (SDLC). However, ML is fundamentally different than traditional applications, and it’s important to account for that in a new, unique process called the machine learning development lifecycle.

    We have identified five challenges every organization should keep in mind as they begin to support ML development.

    Making Heterogeneity Work

    Machine learning is successful when the right tool is selected for a given job. Depending on the use case, a data scientist might choose Python, R, Scala or another language to build one model, and another language for a second model. Within a given programming language, there are numerous frameworks and toolkits available, adding complexity to versioning and consistency.

    For example, TensorFlow, PyTorch and Scikit-learn all work with Python, but each is tuned for specific types of operations, and each outputs a slightly different type of model.

    ML-enabled applications typically call on a pipeline of interconnected models, often written by different teams using different languages and frameworks.

    Figure 1: An ML pipeline that extracts text from scanned documents and analyzes its sentiment passing through several languages and frameworks.

    For example, a public relations firm looking to identify news and reports critical of one of its customers might use the following pipeline:

    1. Extract all text from hundreds of scanned documents with an Optical Character Recognition (OCR) model.
    2. Identify the languages of that text with a language-identification model.
    3. Translate the text to English.
    4. Prepare the text for sentiment analysis with a vectorization model that turns English text to numbers.
    5. Use a sentiment analysis model to score the text.

    Planning for ML Iterating

    In machine learning, your code is only part of a larger ecosystem—the interaction of models with live, often unstructured or unpredictable, data. Interactions with new data can introduce model drift and affect accuracy, requiring constant model evaluating, tuning and retraining.

    Figure 2: The continuous ML Lifecycle with iterations.

    As a result, ML iterations are faster and more frequent than traditional app development workflows. This places greater demands on versioning and other operational processes and requires a greater degree of agility from DevOps processes, tools and staff.

    Organizations interested in adopting ML-enabled applications need to be prepared to handle the burden of iteration without breaking the budget or introducing needless infrastructure tasks.

    Developing the Right Infrastructure

    The model training process typically involves multiple instances of the following:

    • A long and intensive compute cycle.
    • A fixed, inelastic load.
    • A single user.
    • Concurrent experiments on a single model.

    After deployment, models from multiple teams enter a shared production phase characterized by:

    • Short, unpredictable compute bursts.
    • Elastic scaling.
    • Many users consuming many models.

    Operations teams must be able to support both the training and deployment environments on an ongoing basis, because it’s a continuous lifecycle. Selecting the right infrastructure for that job is complicated by a rapidly evolving tech stack, from the increasing variety of processors available for ML workloads to advances in cloud-specific scaling and management, to a never-ending stream of new data science tooling.

    It’s important to future-proof your ML program to ensure a tool or framework you use today does not lead to lock-in or worse—obsolescence—down the road.

    How Will You Scale?

    To address the unpredictability of ML workloads and the premium on low latency, organizations must build compute capacity to support substantial bursts.

    There are three typical architectural approaches, each with benefits and drawbacks:

    Traditional Capacity Planning: In a traditional architecture, operations reserve compute resources capable of scaling to maximum anticipated demand.

     

    Pros:

    • Reserved capacity is always available
    • Easy to administer

     

    Cons:

    • Extreme waste and cost
    • Unanticipated demand can exceed the capacity

     

    Elastic Scaling: Standard elastic scaling designs for a local maximum, scaling machines up and down based on step functions.

     

    Pros:

    • Cost improvements over traditional architecture

     

    Cons:

    • Inefficient hardware utilization
    • Difficult to manage heterogeneous workloads on the same hardware
    • Slightly more management overhead than traditional architecture

    Serverless: Serverless architecture approaches spin up models as requests come in.

     

    Pros:

    • Significantly improved hardware utilization
    • Dramatically lower costs
    • Easier to run and scale different types of workloads on the same machines

    Cons:

    • Substantial administrative overhead for custom-built, on-prem solutions
    • Some third-party services without extensive APIs and reporting provide limited explainability
    • Performance tuning to meet latency

    Architects can implement any of the three approaches in the cloud or in a physical datacenter, though elastic scaling or serverless approaches requires far more effort to manage in-house.

    Trusting the Results with Auditability and Governance

    Explainability—understanding why models make given predictions or classifications—is an essential part of ML infrastructure and the lifecycle. Equally important and often overlooked, however, are the related topics of model auditability and governance—understanding and managing access to models, data and related assets.

    Figure 3: An audit trail can help inform decisions.

    To make sense of complex pipelines, a multitude of users and rapid model and data iteration, deployment systems should attempt to identify:

    • Who called which version of which model.
    • At what time a model was called.
    • Which data the model used.
    • What result was produced.

    Thinking about and planning for these five challenges before beginning a machine learning initiative is a good place to start to ensure a valuable return from your investment.

    To learn more about containerized infrastructure and cloud native technologies, consider coming to KubeCon + CloudNativeCon NA, November 18-21 in San Diego.

    — Diego Oppenheimer