devopsy.kinsta.cloud

Tag: machine learning

  • Why MLOps Hurts: 3 Pain Points for DevOps Teams

    Why MLOps Hurts: 3 Pain Points for DevOps Teams

    Machine learning operations (MLOps) is a practice that aims to improve the collaboration and communication between data scientists and IT professionals in the development, deployment and maintenance of machine learning models.

    MLOps aims to streamline the process of building, testing and deploying machine learning models by automating and standardizing the various tasks involved. This includes automating the training and testing of models and the deployment and management of models in production.

    MLOps involves using tools and processes to automate the various steps involved in machine learning model development and deployment. This includes tools for version control, continuous integration and continuous deployment (CI/CD), and tools for monitoring and managing the performance of machine learning models in production.

    By adopting MLOps practices, organizations can improve the speed and efficiency of their machine learning model development and deployment and reduce the risk of errors and inconsistencies in the process.

    What Role do DevOps Engineers Play in MLOps?

    In MLOps, DevOps engineers play a crucial role in supporting the development and deployment of machine learning models. Their responsibilities may include:

    • Setting up and maintaining the infrastructure needed to support machine learning models, including hardware and software environments, data pipelines, and monitoring systems.
    • Automating the various tasks involved in machine learning model development and deployment, including training, testing, and deployment, using tools such as continuous integration and continuous deployment (CI/CD).
    • Monitoring the performance of machine learning models in production and troubleshooting any issues that may arise.
    • Collaborating with data scientists to understand the requirements and constraints of machine learning models and to help them implement and deploy their models effectively.
    • Managing the release process for machine learning models, including coordinating with different teams, conducting testing and validation, and rolling out updates and improvements.

    Challenges to Implementing MLOps

    MLOps is not without its challenges. Here’s what to consider when developing and deploying your machine learning pipelines. 

    Data Security

    Data security is a key challenge of MLOps for DevOps teams because machine learning models rely on large amounts of data to function effectively. Ensuring the security and integrity of this data is critical to the success of the model and the overall MLOps process.

    There are several risks to data security that DevOps teams need to consider in MLOps:

    • Data breaches: Machine learning models often work with sensitive data, such as personal information or financial data. If this data is not secured properly, it could be accessed by unauthorized parties, leading to data breaches and potential legal and financial consequences.
    • Data tampering: Machine learning models rely on the accuracy and integrity of the data they are trained on. If the data is tampered with or manipulated, it could affect the model’s performance and lead to incorrect or unreliable predictions.
    • Data privacy: Machine learning models may work with data that is subject to privacy regulations, such as the General Data Protection Regulation (GDPR) in the European Union. Ensuring compliance with these regulations is critical to avoid fines and other consequences.

    Model Deployment

    Model deployment is a key challenge of MLOps for DevOps teams because it involves integrating the machine learning model into an existing application or system and setting up the infrastructure to support it. This can be complex, especially if the model is part of a larger application or system, and requires careful planning and coordination.

    There are several challenges that DevOps teams may face when deploying machine learning models:

    • Integration: Machine learning models need to be integrated into an existing application or system in a way that is seamless and does not disrupt the overall functionality. This can be a complex task that requires careful planning and testing.
    • Testing: It is important to thoroughly test the machine learning model before it is deployed to production to ensure that it functions correctly and meets the required performance standards. This may involve testing the model on a variety of data and in different environments to ensure that it is robust and reliable.
    • Performance: Machine learning models may require specialized hardware and software environments to support their performance. Setting up and maintaining these environments can be a challenge for DevOps teams.
    • Scalability: Machine learning models may need to scale up or down depending on the volume of data and the demands placed on them. Ensuring that the model is able to handle different workloads and scale appropriately is a key challenge for DevOps teams.

    Model Performance and Maintenance

    Model performance and maintenance is a key challenge of MLOps for DevOps teams because machine learning models require ongoing monitoring and maintenance to ensure that they continue to perform well in production. This can be a complex and time-consuming task that requires close collaboration between data scientists and IT professionals.

    There are several challenges that DevOps teams may face when it comes to model performance and maintenance:

    • Model drift: Machine learning models can suffer from model drift over time, where the model’s performance begins to degrade due to changes in the data or the business requirements. This can be difficult to detect and can significantly impact the model’s accuracy.
    • Data changes: Machine learning models rely on the accuracy and integrity of the data they are trained on. If the data changes over time, the model may need to be retrained or updated to ensure that it continues to perform well.
    • Hyperparameter tuning: Machine learning models often have several hyperparameters that need to be set to optimize their performance. Identifying the optimal values for these hyperparameters can be challenging and the model may need to be retuned over time to maintain its performance.
    • Model versioning: Machine learning models may need to be updated or replaced over time to improve their performance or to address issues. Managing different versions of the model and coordinating the rollout of updates can be a challenge for DevOps teams.

    5 MLOps Best Practices to Help DevOps Teams

    There are several best practices that DevOps teams can adopt to help overcome the challenges of MLOps:

    1. Automate as much as possible: Automating tasks such as model training, testing and deployment can help streamline the MLOps process and reduce the risk of errors and inconsistencies.
    2. Use version control: Version control systems such as Git can help track changes to the machine learning model and its supporting infrastructure, making it easier to roll back changes or deploy updates.
    3. Use CI/CD: CI/CD pipelines can help automate the process of building, testing, and deploying machine learning models, improving the speed and efficiency of the MLOps process.
    4. Monitor and log model performance: Setting up monitoring and logging systems can help DevOps teams track the performance of machine learning models in production and identify any issues that may arise.
    5. Collaborate and communicate effectively: MLOps requires close collaboration between data scientists and IT professionals. Establishing clear communication channels and effective collaboration processes can help ensure that all parties are working towards a common goal.

    Conclusion

    In conclusion, MLOps can be a challenging process for DevOps teams due to the complexity and specialized nature of machine learning models. There are several pain points that DevOps teams may face when implementing MLOps, including the complexity of setting up and maintaining the necessary infrastructure, the difficulty of collaborating and communicating effectively with data scientists and the challenges of ensuring the security and integrity of data used to train and test machine learning models. 

    By adopting best practices such as automation, version control, CI/CD pipelines and effective collaboration and communication, DevOps teams can help overcome these challenges and improve the speed and efficiency of machine learning model development and deployment.

  • Navigating the MLOps Revolution: Three Missteps to Avoid

    Navigating the MLOps Revolution: Three Missteps to Avoid

    With nearly 60% of businesses set to adopt MLOps by 2024, this function has quickly established itself as an indispensable part of the enterprise technology world. And with computing needs rapidly expanding as digital transformation efforts continue, the growth of the MLOps market is unlikely to stop anytime soon.

    That said, as with any emerging technology–particularly something as powerful as MLOps–there is no shortage of potential pitfalls that businesses and their tech teams can fall into if they move too quickly. More importantly, these missteps can have significant impacts on a company’s ability to generate revenue and optimize performance.

    With that in mind, below are three of the biggest mistakes that companies tend to make when adopting MLOps.

    Jumping in Without Testing the Waters

    If you doubt your MLOps maturity, you should probably wait. One of the biggest factors that undermine MLOps adoption and long-term success is that companies jump in before they are actually ready to adopt the technology. This not only leads to MLOps feeling like a burden but also often devolves the customer-vendor relationship into a transactional sales relationship rather than what it should be: a partnership. Onboarding technology needs to be a give-and-take process with ongoing dialogue between the customer and the vendor. If not, vendors – even the most ethical ones – have to rely on partial situational intelligence and guesswork to offer solutions that may or may not address the problems and deliver results. The best way to overcome this is for companies to map out exactly what MLOps challenges they are up against, what barriers are present, and also the outcomes they are looking for. With these key components understood and agreed to in advance, not only can companies make sure they are getting the best vendor partner, but the vendor can then provide much more tailored guidance.

    Misunderstanding What MLOps Actually Is

    There is a tendency among new adopters to distill MLOps down to the idea that it is made up of two components: Storage and compute. Yes, these are two significant elements of the MLOps equation, but this view is a massive oversimplification. Also, if it is not clarified at the outset, this misperception can lead organizations to feel overwhelmed by MLOps and make adoption less “sticky” as a result.

    The fact is, MLOps is composed of numerous pieces–from experimentation to deployment–that each come with their own specific nuances. Each of these parts is interconnected to the next and can derail MLOps results if not properly planned for and maintained. In addition, making sure that every necessary component is functioning the way it should in order to meet the needs and expectations of each client takes time. So businesses and their broader DevOps teams need to be prepared to exercise patience if they have any hopes of successfully adopting MLOps in a way that is geared towards long-term sustainable results.

    Not Realizing the Amount of Iteration MLOps Takes

    When looking at projects from a traditional DevOps perspective, there is a common belief that if a project is stable enough to get it to DevOps, it is ready for production. With MLOps projects, however, this simply isn’t the case. In comparison to DevOps, MLOps is prone to evolution – meaning that iteration and reiteration are incredibly common. For example, in the midst of the MLOps process, companies will want to iterate over the existing process, sometimes at a much more frequent cadence than their software release cadence. This means that there is greater ongoing contact and collaboration between MLOps teams and other teams. This interaction stands in stark contrast to traditional DevOps projects where once production is complete, developers hand over the keys and are essentially hands-off. Businesses and their traditional DevOps teams need to become comfortable with this new way of doing business and build in the flexibility and leeway to make sure that while their MLOps is able to function correctly and grow, that it does so in a way that works for them.

    Businesses are right to be excited about the prospect of adopting MLOps. However, to hit the ground running and deliver impact over the short, medium, and long term, it is essential that companies deploy a comprehensive strategy to ease onboarding headaches and make the process as seamless as possible. And by keeping these key missteps in mind, businesses can do just that.

  • Finding Value in AI-Augmented DevOps

    Finding Value in AI-Augmented DevOps

    While artificial intelligence (AI) and machine learning (ML) are emerging technologies, we know they can help an organization parse large data sets and glean actionable insights. But do AI-infused processes actually make a difference in organizations that employ DevOps? The answer is explored in a recent survey by Tricentis. Working in partnership with Techstrong Group, Tricentis surveyed more than 2,600 DevOps practitioners across the globe about their AI-augmented DevOps processes. The results pointed to a bright future in DevOps.

    In this survey, the majority of respondents cited they’re already getting value from AI-powered DevOps processes. A third of survey respondents found AI-augmented DevOps extremely useful today and nearly half found it very useful. Although the technology is still relatively young, a 2021 report by IT infrastructure automation company Puppet Labs found that 83% of organizations are implementing DevOps processes. This rapid adoption speaks to the promise of the technology.

    Interestingly, nearly two-thirds of respondents cited that testing is the area within DevOps where AI will have the most impact. Testing is a major pain point for DevOps organizations because a scaled DevOps environment calls for complex testing scenarios and large amounts of data, and many organizations struggle to scale their automation to the level DevOps requires. Without automation, manual testing can significantly delay releases. In AI-infused DevOps processes, AI can help accelerate the authoring of tests, help testers understand where the risks are, and repair broken tests.

    AI’s Greatest Potential is in Functional Testing

    We typically think of software testing as either functional, where the program enables users to properly execute tasks, or non-functional, where the application is secure, fast and stable. Respondents were asked which disciplines of testing stood to gain most from AI and 65% of them indicated functional testing. Unit testing and UI testing were the sub-categories most often selected.

    The complexity of the functional testing process is where AI shines. Functional testing is open-ended and many permutations are required to ensure proper test coverage. AI steps in and manages those permutations, parsing large amounts of data and providing insights. Additionally, AI can help wrangle production signals that aren’t discovered in a test environment.

    Throughout the survey, respondents specifically noted AI’s promise in regard to UI testing. UI testing is perhaps the most time-intensive and manual type of testing in most organizations. With the many variations of the user experience paths that exist, AI can help mimic the actions of a real end-user, adjusting for application nuances or variations in experience. AI and ML can also boost your build process, increasing the efficiency of repetitive tasks. The ability to quickly identify and resolve issues can improve the effectiveness of your automation.

    While survey respondents see the most potential in testing, they acknowledge that AI-powered DevOps can address other business challenges. More than 40% claim that these processes can reduce the skills gap and empower junior staff to perform complex tasks, improve the customer experience, cut costs, boost innovation and increase efficiency among developers. Nearly 40% also believe that AI-powered DevOps delivers leadership the insights they need for continuous improvement.

    Automation is the Key to Scaling DevOps

    It’s worth noting that among the DevOps organizations that see benefits from AI and ML, most of them are mature organizations; they employ DevOps workflow pipelines, toolchains, automation and cloud technologies, and they’ve automated more than half of their testing. Only 21% of practitioners who are just dipping their toes into DevOps have achieved this level of automation. And even in mature organizations, testing is still the main bottleneck; only 40% of them have automated more than half of their testing.

    The key to scaling DevOps practices is developing test automation that can keep up with an increasing volume and speed of releases. But the road to DevOps isn’t always easy. Practitioners cited obstacles like a lack of tech skills (44%), insufficient budget (25%) and tool selection (19%). Organizations that want to implement DevOps need to rely on external vendors and internal resources to address these challenges and effectively scale.

    Looking to Adapt More DevOps Practices?

    So where does an organization that wants to implement DevOps start? With the big questions. It’s crucial to identify the specific business problem that you’d like to solve. What’s the gain you’re looking for? Where in the software life cycle do you expect to find a benefit? AI and ML can add business value, but they don’t solve every problem across the life cycle.

    After an organization has identified its DevOps goals, its focus should shift to automation. To reap the benefits of AI-powered testing, an organization should automate repetitive functional testing, especially unit and regression testing. Most organizations start by implementing continuous integration and continuous delivery. Once an organization has taken these steps, it will start to amass a large cache of results and artifacts generated from running the test cases, which AI can mine to increase test stability and identify recurring problems. The more you automate, the more you’ll get out of it and the faster you’ll accelerate your delivery.

    Organizations just starting out with DevOps should also understand the distinction between artificial intelligence and ML. Most people refer to ML as AI, but they’re not interchangeable, and most organizations haven’t advanced to the use of AI. A true AI is broadly powerful, while ML technologies are narrow and solve a specific domain problem that a model was trained for.

    When you’re evaluating potential vendors, there are a number of key questions to ask, including:

    • What specific AI technology is the vendor using? Is it an off-the-shelf library or something they built themselves?

    • How are they using AI? Which algorithms are in play and what approach do they take?

    • How do they train their models? Different types of patterns emerge in data sets depending on the industry, the frequency with which data is updated and the application. The data population that the vendor is training on must look like your production data, or the model could take you in the wrong direction.

    • How do they ensure that bias doesn’t come into play? If they train models in a different domain or vertical, for example, you could be an outlier for that vendor.

    • How do they handle failure? How do they monitor for miscategorization and misapplication of the AI in a way that lets you trust the data?

    • Will your data be used to train and improve your learning models? If so, does that raise any privacy concerns? Expectations around data privacy should be explicit and leave no questions.

    • Are there other customers in your vertical with a similar business profile that are getting the benefits you’re looking for? Can the vendor provide examples?

    What Does AI-Infused DevOps Look Like in Practice?

    Consider a financial services organization that has more than 20 applications to manage. With so many applications, it’s necessary to conduct significant UI testing and the tests must be stable. One of the biggest challenges to UI testing lies in locating various web elements; most people start with manual script-based automation, but it can be challenging and time-consuming to create and maintain. Employing AI and smart locators to identify these elements can save time and drastically cut down on UI testing maintenance.

    Survey respondents were asked how they’re currently using AI to augment their testing processes. Thirty-seven percent say they’re accelerating the creation of their automated test cases, while 44% focus their testing on the areas of highest risk. Almost half are able to reduce test case maintenance with self-healing and 43% use AI to identify the root cause of failed tests. Thirty-one percent claim that AI offers insights into test process improvements and 34% use AI to help identify tests to run based on changes within the application.

    Where is AI-Powered DevOps Going Next?

    Going forward, AI will become more and more embedded in testing processes. Great progress with off-the-shelf vendors is being made, and their products are a lot more accessible than they used to be. As the tooling continues to mature, these products will reduce the need for specialized skills. Internal initiatives like data lakes or data warehouses will persist, but there will be less need to build large data science teams to get the advantages of AI.

  • Stable Diffusion Goes Public — and the Internet Freaks Out

    Stable Diffusion Goes Public — and the Internet Freaks Out

    Welcome to The Long View—where we peruse the news of the week and strip it to the essentials. Let’s work out what really matters. (more…)

  • MLOps Vs. DevOps: What’s the Difference?

    MLOps Vs. DevOps: What’s the Difference?

    Machine learning operations, or MLOps for short, is a key aspect of machine learning (ML) engineering that focuses on simplifying and accelerating the process of delivering ML models to production and maintaining and monitoring them. MLOps involves collaboration between different teams including data scientists, DevOps engineers, IT specialists and others.

    MLOps can help organizations create and improve the quality of their AI and machine learning solutions. Adopting MLOps allows machine learning engineers and data scientists to collaborate to improve model performance by implementing continuous integration and continuous deployment (CI/CD) practices. It accelerates the ML model development process by incorporating the appropriate monitoring, governance, and validation of ML models.

    What Is DevOps?

    DevOps combines the concepts of development and operations to describe a collaborative approach to performing the tasks usually associated with separate application development and IT operations teams. DevOps, in its broadest sense, is a philosophy that encourages improved communication and cooperation between these (and other) teams within an organization. 

    In its narrowest sense, DevOps refers to adopting practices that enable the deployment and maintenance of iterative application development, automation, and programmable infrastructure. It also includes changes in the workplace culture, like trust-building and bonding between developers, system administrators and other team members. DevOps aligns technology with business objectives and can transform the software delivery chain, job functions, services, tools and best practices.

    MLOps Vs. DevOps: Key Differences

    Here are some of the main differences between MLOps and traditional DevOps.

    Development

    The concept of development refers to different things in each model, with a slightly different CI/CD pipeline.

    DevOps:

    • Usually, the code creates an interface or application. 
    • The code is wrapped into an executable or artifact before being deployed and tested with a set of checks. 
    • Ideally, this automated cycle will continue until the final product is ready.

    MLOps:

    • The code enables the team to build or train machine learning models. 
    • The output artifacts include serialized files that can receive data inputs to generate inferences. 
    • Validation involves checking the trained model’s performance based on the test data.
    • This cycle should also continue until the model reaches a specified performance threshold.

    Version Control

    DevOps:

    • Version control typically only tracks changes to code and artifacts.
    • There are few metrics to track.

    MLOps: 

    • MLOps pipelines usually have more factors to track. Building and training an ML model involves an iterative experimentation cycle, requiring tracking of various metrics and components for each experiment (essential for later audits). 
    • Additional components to track include training datasets, model building code and model artifacts. 
    • Metrics include hyperparameters and model performance indicators, such as error rates.

    Reusability

    DevOps:

    • DevOps pipelines focus on repeatable processes.
    • Teams can mix and match processes without following a specific workflow.

    MLOps: 

    • MLOps pipelines repeatedly apply the same workflows. The common framework across projects helps improve consistency and allows teams to progress faster because they start with familiar processes. 
    • Project templates offer structure, enabling customization to address the unique requirements of each use case.
    • Uses centralized data management to consolidate the organization’s data to accelerate the discovery and training processes. Common approaches to centralization include a single source of truth and data warehouses.  

    Continuous Monitoring

    Monitoring is essential for both DevOps and MLOps, but for slightly different reasons.

    DevOps:

    • Site reliability engineering (SRE) has been trending over the past few years, emphasizing the need for monitoring software from development through to production deployment. 
    • The software does not degrade in the way an ML model does. 

    MLOps:

    • Machine learning models can degrade quickly, requiring constant monitoring and updating.
    • Conditions in the production environment affect the model’s accuracy. After deployment to production, the model starts generating predictions based on new data from the real world. This data is constantly changing and adapting, reducing model performance. 
    • MLOps ensures that algorithms remain production-ready by incorporating procedures to facilitate continuous monitoring and model retraining.

    Infrastructure

    DevOps and MLOps both rely heavily on cloud technology, but have different operational requirements.

    DevOps relies on infrastructure such as:

    • Infrastructure-as-code (IaC)
    • Build servers
    • CI/CD automation tools

    MLOps relies on infrastructure such as:

    • Deep learning and machine learning frameworks
    • Cloud storage for large datasets
    • GPUs for deep learning and computationally-intensive ML models

    DevOps and MLOps Trends

    Here are some of the major trends driving the development of DevOps and MLOps.

    GitOps

    A new evolution of the DevOps workflow, GitOps is a new paradigm for controlling and automating infrastructure. A Kubernetes-oriented paradigm enables developers and operations teams to manage Kubernetes clusters and deliver containerized applications using Git. Implementing Git workflows for operations and development teams allows developers to leverage Git pull requests to manage software deployments and infrastructure.

    GitOps incorporates existing development tools to manage cloud-native and cluster-based applications with CI/CD. It automatically deploys, monitors, and maintains cloud-native applications using a Git repository as the single source of truth.

    GitOps is a way to implement and maintain clusters in Kubernetes. Continuous delivery and deployment allow developers to build, test, and deploy software faster through incremental releases. Kubernetes continuous integration and runtime pipelines must be able to read and write files, update container repositories, and load containers from Git. GitOps helps businesses manage their infrastructure with version control, real-time monitoring, and alerting of configuration changes.

    Synthetic Data

    Synthetic data is any information generated artificially instead of collected from real events. Algorithms generate synthetic data for use as a replacement for operational and production test datasets. Synthetic datasets are also useful for validating mathematical models and training machine learning models.

    Benefits of synthetic data include:

    • Minimizing the constraints associated with using sensitive and regulated data.
    • Customizing data to specific requirements and conditions not available in real-world data.
    • Generating data for testing software quality and performance for DevOps teams.

    Codeless Machine Learning and AI

    Machine learning often involves computer code to set up and process model training, but this is not always the case. Codeless machine learning is a programming approach that eliminates the need for ML applications to go through time-consuming processes.

    Codeless ML eliminates the need for experts to develop system software. It is also simpler and cheaper to deploy and implement. Using drag-and-drop input during machine learning processes can simplify training efforts in the following ways:

    • Evaluating results.
    • Dragging and dropping training data.
    • Creating predictive reports.
    • Using plain-text queries.

    Codeless ML gives developers easy access to machine learning applications, but it is no substitute for an advanced, nuanced project. This approach is suitable for small businesses that lack the capital to maintain an in-house data science team.

    TinyML 

    TinyML is a new approach to machine learning and AI model development. It involves running models on devices with hardware restraints, such as microcontrollers powering smart vehicles, refrigerators and electric meters. This strategy is the best fit for these use cases because it speeds up the algorithm—there is no need for the data to go back and forth to a server. It is especially important on large servers and can speed up the entire ML development process.

    Running TinyML programs on an IoT edge device has many benefits:

    • Lower energy consumption.
    • Reduced latency.
    • User privacy guarantees.
    • Reduced bandwidth requirements.

    Using TinyML offers greater privacy because the computation process is entirely local. It consumes less power and bandwidth, resulting in lower latency because it doesn’t require sending data to a central location for processing. Industries that are taking advantage of this innovation include agriculture and healthcare. They typically use IoT devices embedded with a TinyML algorithm to monitor and predict real-world events using collected data.

    Conclusion

    In this article I covered the key differences between MLOps and DevOps:

    • Development—DevOps pipelines focus on developing a new version of a software product, while MLOps focuses on delivering a working machine learning model.
    • Version control—DevOps is mainly concerned with tracking binaries and software artifacts, while MLOps tracks additional factors like hyperparameters and model performance.
    • Reusability—DevOps and MLOps both strive to create reusable processes and pipelines, but use different strategies to achieve repeatability. 
    • Continuous monitoring—Monitoring is important to DevOps, but even more important in MLOps because models degrade in performance due to model and data drift.

    Finally, I covered a few key trends that will transform DevOps and MLOps in the near future. I hope this will be useful as you discover your place in the new and exciting development ecosystem.

  • DevOps Unbound EP 19 – How AI and ML are being used in DevOps Today – TechStrong TV

    DevOps Unbound EP 19 – How AI and ML are being used in DevOps Today – TechStrong TV

    Alan Shimel: Hey, everyone. I’m Alan Shimel, CEO of MediaOps, devopsy.kinsta.cloud, Container Journal, Security Boulevard, and you’re watching DevOps Unbound. DevOps Unbound is sponsored by our friends at Tricentis. So many thanks to them. And DevOps Unbound is a biweekly show where we cover topics of interest to the DevOps audience. I am the host. My co-host is our CTO and CEO of accelerated strategies group, my friend Mitchell Ashley. Mitchell, welcome.

    Mitchell Ashley: Thank you. Good to be here as always, and with this illustrious panel.

    Alan Shimel: Absolutely. And an illustrious panel, it is. We have two panel members joining Mitchell and I today, let me introduce you to both of them. They’re both great folks in their own and I’m gonna let them introduce themselves. Let’s start with our friend Judith Horowitz. Judith, welcome to DevOps Unbound.

    Judith Hurwitz: Thank you so much, Alan. It’s a pleasure and honor to be here.

    Alan Shimel: It’sour pleasure.

    Judith Hurwitz: So I’m Judith Hurwitz. I’m the co-author of 10 books. Have been in the industry for 30-plus years. Focus on everything from DevOps, security, manageability in cloud, hybrid cloud, and really looking at how you take technology to transform organizations. It’s a complex topic, not easy, but it’s what we’re in the middle of.

    Alan Shimel: Absolutely. Then last, but certainly not least, my friend Brian Dawson. Hey Brian, welcome.

    Brian Dawson: Hey, Alan. Thank you. Good to be on with you again. To tell the audience a bit about myself. As I’ve told you before, Alan, I’ve been in software development and delivery for about 30 years. I consider myself a technologist and even prior to focusing on DevOps, I’ve had a focus on optimizing software development and delivery.

    Excited to talk about AI. I kind of dabbled and dipped my toes in the space and my time that I spent at the company that is now PlayStation. Things have come a long way since, and during that time, I’ve spent the past 10 years of my career focused on identifying, applying and spreading DevOps practices. So I’m excited to discuss the two together here with this group.

    Alan Shimel: Absolutely.

    Participant: Brian, I’m not sure if you mentioned your present position with the Linux Foundation.

    Brian Dawson: I did not actually, because also this organization I’m at today has a big footprint in the AI and ML space. So today I’m with the Linux Foundation, I oversee our developer relations in ecosystem development. And of note within that is the LFAI & Data Foundation which houses a number of impactful products in this space projects. Excuse me.

    Participant: Thank you. I just thought the folks at LF would welcome that.

    [Crosstalk].

    Brian Dawson: Well, appreciate it. Thank you.

    Participant: Yep

    Alan Shimel: All right. So the topic for today’s DevOps Unbound is artificial intelligence and machine learning, how can they help improve performance of DevOps teams? In other words, what’s their role in the DevOps world? And let me preface our conversation by giving you sort of my view of it.

    DevOps for many people was all about automation. What can we automate? Let’s automate everything we can, automate, automate, automate. And obviously in discussions around automation, topics such as artificial intelligence and machine learning, you would tend to think help with automation. And therefore anything that helps with automation is good for DevOps.

    And I should also say that we lump AI and ML together, like their siamese twins joined at the hip, and they’re not necessarily conjoined. You can have AI without ML and you can have ML without AI, at least I think so. I’d be interested in your thoughts.

    But has it lived up to the hype? Will it ever live up to the hype? Was the hype unfounded? Was it unrealistic? What role has AI and/or ML to this point played on DevOps and what role will it play going forward? I think that’s our topic today. I mean, any one of you, if you want to kick it off with your own feelings or respond, go right ahead.

    Brian Dawson: Well, I’d like to jump in as a troublemaker early on and say –

    Alan Shimel: Okay.

    Brian Dawson: – I challenge Alan that DevOps is really about automation is kind of the basis for the tie. If we look at it, DevOps is really a set of cultural practices and tenants that align DevOps and other software delivery stakeholders around the shared objective of delivering quality software reliably, rapidly, and repeatedly.

    Now the reliably, rapidly and repeatedly component, absolutely lends itself to some of the benefits that we could mine from AI. But I actually think it’s a great opportunity if we don’t say DevOps equals automation equals AI, but rather we say DevOps equals a culture aligned around a shared objective and how can AI and ML help support those shared objectives.

    Judith Hurwitz: So I think you make a great point. Brian, I think one of the most important issues, and you mentioned culture. We have been, for what, 50 years through this development and operations perspective on how we get things done. And the promise of AI is, “Okay, can push a button and it will take care of everything.” And, and this promise has been there for, what, 20 years.

    The reality is, it’s just not that simple. There are definitely things that we are doing today, and we are seeing evolve that is definitely helping the developers and the operational professionals. For example, if you have repeatable functions that happen all the time, that for example, if you press a button and a certain task should happen, you can probably use a model and collecting massive amounts of data to do that automatically.

    nd that’s a very good use of AI, and will work today. They call it ML Ops or AI Ops, and there’s a good reason why that’s what we’re focused on today, is because you’re look for predictable patterns and predictable anomalies so you can avoid making stupid mistakes that, “Oh my God, why did I do that? I knew what the right answer was.”

    But AI is not a panacea. And I think if we look back even two, three years ago, you had companies that all were saying, “Okay, we have automated, we’ve put AI into DevOps and all you have to do is press button and all of your problems are over.” That’s just not reality.

    Alan Shimel: Yeah.

    Mitchell Ashley: Yeah, there’s the marketing AI and ML, and then there’s real AI and ML. Marketing, meaning everybody glams onto the term and uses it, meaning it’s a case statement or an if statement in people’s logic in their software. I started doing some work in AI in the 80s and 90s programming and Lisp prologue, and doing some corporate education in that area, some expert systems with triage, things like that.

    So dabbled in a little bit there. And then I was exposed to more recently, about five, six years ago, worked with a gentleman named Dr. Bernardo Huberman, who’s one of the experts in the field. And I asked him a me understand why is machine learning taken off as this sort of, he described it as a subset of AI. So you can know that was his model anyway.

    What he told me really made sense, and it kind of helps me understand where can we apply machine learning? And that being kind of the most popular part of AI. And he said, “Machine learning takes massive amounts of data.” And the fact that we have the cloud and all these applications, creating data, a lot of data exhaust. Now, people are going back and mining that data.

    So we have this whole, Judith, in your writing, in your books about data analysis and leading up to AI, machine learning is great for that, because you can have supervised or unsupervised algorithm.Supervised, meaning that’s a cat, that’s a cat, that’s not a cat, or unsupervised, which is gonna pour through the data and start to look for those patterns and trends and anomalies that you were talking about Judith.

    So when people talk about using AI or ML in our industry, I always think about, well, where’s there lots of data, and can we leverage it in some impactful way? So as I look at products or technologies that claim they’re doing that, that’s at least one sort of criteria to fair it out. Is it real there? Is there something real there, or is it more spin and fluff to help us? We all have case statements in our spot, right?

    Alan Shimel: Absolutely. Actually, Brian, it looked like you were gonna say something, I didn’t wanna jump on you.

    Brian Dawson: No, Alan. I mean, of course I have a lot to say, but nothing in particular, I just, since I’m off mute, I’ll say I absolutely agree with Mitch and Judith and underscore the points that they’ve made.

    Alan Shimel: Yep. You know what, I wanna take a moment though, and explore, Brian, you disagree DevOps isn’t about automation. And I get the whole cultural aspects of DevOps and all of that, but certainly automating as a way to be more efficient, to get more done faster, I think is part and parcel of the DevOps mindset.

    Brian Dawson: Yeah. And if I may and without waiting to see where you wanted to go with that, I’d say, no, it, it is absolutely important, but there’s a line Jez Humble would use in talking about CD. And frankly, I think the book that Dave Farley, Jez Humble and team, Road Around Continuous Delivery is undercelebrated and under referenced.

    And in talking about that, one of the things he said is continuous delivery. And I’m gonna say by extension, some aspects of DevOps don’t require any tools. I can do continuous delivery with a bash script.Catch is it’s not the most efficient in pursuit of effective collaboration of delivery velocity so you can iterate. And as we’ve talked about in this past, establish that control feedback loop that requires things like automation to achieve those goals.

    I just think sometimes when we talk about what we see going on with AI and ML, and in those terms, we overload a principle or practice with expectations that are core to it. And I worry a bit about that, so I wanna call it out. And sorry, Alan, I don’t know if that’s where you wanted to go with the commentary.

    Alan Shimel: No, no, no. I wanna go wherever you wanna go, Brian. It’s a collective. Judith, what about – I’m sorry, go ahead. Oh

    Judith Hurwitz: Oh, yeah. So,Alan, I think one of the key issues is how the whole area of continuous delivery. In previous generations, you would build an application and it would live pretty much as it was written, could be for 10 years. Today, applications are constantly having to be revised because customers change, partners change, the sources of data change.

    So, unless you have the ability to constantly update and change and modify, you lose out. One of the values of using automation and using AI and machine learning models with lots of data, is to support the ability to do this. You don’t have enough hands and enough brains to where problems may occur, because you’ve changed things.

    How many times have we seen problems occur because somebody has added a new service to their environment and somebody forgot to change a configuration file? Now that’s not something that you need a massive brain to be able to do, but people get busy and they forget to do simple things.

    Mitchell Ashley: I think, Judith, that’s really a good point, ’cause I wanted to ask you about this in the software creation process. Given things are so dynamic now and constantly changing, understanding just the environment, the infrastructure’s code all the way up through the application and how much all of that is changing, it seems like the stepping in for humans in certain conditions in that environment is a great application of AI.

    For someone to synthesize all the factors that might go into where a problem exists or what might be causing a problem, seems to be where some of those algorithms might be helpful. Do you agree with that? Has that been your experience of what people are thinking about for AI?

    Judith Hurwitz: Yeah, definitely. I think what we’re seeing is this is where sort of the human factor comes in. You set it up so that if the printers turned off, don’t send me an alarm fast emergency. There’s a problem with the printer. We all have gotten used to that. But when there are problems that you you’ve never seen before that you don’t have data on and the model doesn’t take into account, then the AI Ops or ML Ops then gives you a message, “There’s something strange going on. I don’t know how to fix it. What do you want do here?”

    And so over time, when that appears again, well now you had data, well, this occurred once before, maybe it was anomaly twice. But then when you have enough data and enough experience over time, then you build that into the model so the next time that occurs, you make a fix.

    On the other hand, you don’t want the system to say, “Oh, I know what that is, make a fix,” when it turns out, no, no, no, you don’t understand the context of, of what’s going on here. And just because there’s a correlation, doesn’t mean that there’s a cause for that.

    Mitchell Ashley: It’s good to know, ’cause it’s not always about automating a response to it. I know in the security world, we’re very cautious, we’re skeptical about those things happening, blocking, legitimate traffic and financial world. That’s a huge issue that we run into.

    So you pointed out a good scenario, whereas we see repetitive patterns over time and that can help machine learning algorithms now understand, “Okay, that’s what that pattern looks like,” so you can identify what it is next time instead of it’s just an anomaly.

    Brian Dawson: Yeah. It’s interesting Mitch that you say that, and going back to the original, kind of the starting topic about AI and ML, not really achieving, its its promise yet. And as you called out, getting data is an issue, it’s not necessarily applicable to every space. You need a level of determinism and predictable patterns that you can learn from and build on.

    And I have for a long time been excited that when we look at CI and CD, when we look at, Alan, the automated workflow of software development and delivery, or even if it’s not automated kind of the standard, there’s a couple of things that you do have. You’re doing builds, sometimes thousands of times a day, builds in deployment, getting that out into pro. You could be generating a ton of data.

    It’s also in its nature, you’re striving to achieve a level of consistency and repeatability in how you deliver software. And I’ll say I’m not an expert to the level. Some of the people on this episode are, but I get at a very sort of top level get really excited about the opportunity for AI and ML to help better CI and CD, and in line with the principles of DevOps.

    What I see it really helping with is reducing cognitive loads so developers can focus on coding, innovating and solving problems while helping ensure that quality and stability that’s difficult to maintain while you’re moving fast. So just thought I’d called out.

    And I am curious to see when vendors and others really start to do like what our friend Koski has done with Launchable, and really apply AI and ML to the left hand side or dev side of the process to reduce the load, insure quality and stability

    Alan Shimel: Guys, we’re all lumping AI and ML, AI and ML, AI, and ML. Does it have to be AI and ML? Isn’t it, in my experience, a lot of what we call AI and ML is a lot more on the ML than it is on the AI? So, is it fair to call it truly AI or is it really ML? And maybe ML is for today and AI is it tomorrow. I don’t know.

    Judith Hurwitz: So I think there’s definitely a problem of nomenclature here. What we’re really dealing with primarily now is models and modeling data and creating models from data. That’s the reality of today. I think that there is a lot of misuse of the term AI and there have been some absolutely wild predictions. I can’t remember the name of the computer scientist who predicted that he would be able to replicate the human brain with AI.

    Mitchell Ashley: Marvin Minsky treat.

    Judith Hurwitz: No, this was past Minsky.

    Mitchell Ashley: Oh, was it after? Okay.

    Judith Hurwitz: Yeah, no, I mean, this was in the last five years.

    Mitchell Ashley: Oh, okay.

    Judith Hurwitz: So, over time, you’re always gonna be dealing with models. You’re never gonna get rid of the models. AI is sort of the next – Models are a subset of what you eventually achieve with AI, but AI is really a concept that will probably take decades to evolve. And I think one of the problems we have right now is where you have vendors, and I’ve talked to hundreds of them over the years where they say, “We have an AI application.”

    It’s because it’s a hot buzzword and you can look back and in the history over the last 30 to 40 years and see whatever the hot topic is, all of the vendors say we do that. So I think that that’s one of the problems that businesses are facing right now. What is the difference? What does it really do? How does it make your company better? How do you use this technology to be prepared for change?

    Mitchell Ashley: I think that’s a good way to break it down too, because early days of AI were about emulating human thinking. We’re still long from – I’m not sure – All my intelligence is artificial, by the way, I acquired all of it. I don’t think I was born with any of it. But that was kinda where it started. And then AI became about expert systems largely, and then machine algorithms.

    Machine learning algorithms was really where I think most of the activity is today because of that prevalence of data. It seems like most organizations are faced with sort of one of three strategies, what do we look from our vendors and how they may use AI in a meaningful way, or machine learning in a meaningful way that’s gonna help my business.

    Do I build models myself from the data like you were talking about, Judith, to apply to like a financial analysis kind of situation, more of an expert system, or do I use machine learning algorithms in my own software to do interesting, valuable things that my software can do.

    And maybe you play in all three of those or a subset of those, but that seems to be the question I think most individuals or organizations are at. We’re not at a place where we can go build models, but we’re looking for these capabilities either in our own code or in third party products.

    Judith Hurwitz: So I heard an interesting story from one of my clients a few years ago, maybe five years ago. He had a client that was very gungho about AI. So he went out and he hired five data scientists, paid them a million dollars each, gave them their own space and left them alone. These are the smartest people on the planet. Came back in six months. Didn’t wanna bother them. They’re so smart.

    “All right. What have you found out? Have you written this application?” And they said, “Well, we’ve been discussing this for six months and we have determined what algorithm we’re going to use. The point is, that they worked in isolation. They thought they were the smartest people on the planet. They did not talk to people on the business side about what business problems they had.

    They didn’t talk to the people who understand corporate data. They didn’t ask them, “What data do you actually have? What data do you need?” They didn’t talk to people who knew the business strategy or the business processes that were in place or needed to change. So it really is a team sport.

    And I like Brian’s discussion in the beginning about culture, because it really is about sort of hybrid organizations where you have people, you have leaders that know a little bit about all of these areas, and then a team that’s brought together that can work across these areas.

    Mitchell Ashley: Sure.

    Brian Dawson: Judah, can I ask, do you have any recommendations for DevOps teams on how do we evaluate or investigate where ML is a solution? And I ask this based on observation, we’ll see if we agree, that oftentimes people are looking for a problem to apply the solution to, as opposed to looking for ML as a solution to a problem they’ve identified. I’m just curious if you [crosstalk].

    Judith Hurwitz: I violently agree with you.

    Brian Dawson: Okay.

    Judith Hurwitz: So I think, and a lot of times people get so enamored with a new technology that they look at it as a way to solve all problems.And for a DevOps team, I think we are finally getting to the stage where there’s really reality in DevOps. It’s not the developers who are saying, “That’s not my problems. The operations people have to make this work.”

    There’s really beginning to be this collaboration between development and operations. And this shift left is definitely becoming real. So that’s definitely true, but for these teams to be successful, they have to have a holistic view of, where is your business going? What do they actually need? Why have they come to you and say, “We gotta do something?”

    Is it just because it would be cool to do something and spend money,or is there a real rationale behind this that they need from? What’s the pain that’s out there that they can solve? So, they have to start with that. They have to start not only collaborating between developers and the operations team, but with the business leaders, with the people who understand all of the data, people who understand security. So, all of this comes together in very much a holistic pattern.

    Alan Shimel: Fair, fair. Hey guys, we’re way past halfway through here and I wanted to kind of turn our conversation to the future. We’ve had a discussion on sort of the history of AI and ML and lessons learned, et cetera. But when we look forward, Brian, you mentioned the Linux Foundation has a, I dunno if it’s a daughter foundation or a subgroup, dedicated to AI and technologies.

    What do we see when we look sort of near term future? Forget, yes, one day, we’ll mimic human brain patterns, who knows if we’ll be alive by that, but near term, what do you see? What is Linux Foundation planning for?

    Brian Dawson: Well, looking forward, I – And it’s funny, ’cause you brought me in and I started to think about the impossible dream of what’s gonna happen in the future. I will call out and will mention again, I said at the start, the Linux Foundation is a parent foundation of what we call a Linux Foundation open source project that hosts other project.

    So LFAI & Data, you can call it a sub-foundation of Linux Foundation because they host, I believe at this point, 12 graduated projects with about 20 to 30 projects in incubation. So we’re talking 30 to 50 projects under the LFAI & Data umbrella that are all working on various aspects of shared efforts, multiple, large commercial organizations, as well as standards bodies coming together to drive the future of of AI and ML.

    Brian Dawson: What I do see coming out of LF AI & Data Foundation in the short-term is standards and foundational implementations, i.e. moving sort of beyond the sort of rudimentary discovery around AI, building sort of packaged or gray box implementations that everybody agrees and collaborates on, which I think will help unlock the less initiated, less expert vendors to begin to deliver truly AI based differentiating capabilities.

    So, to put that in short, because there was a lot of words is, I think it’s about establishing a foundation for us to start to build on and accelerate our progress within the AI and data space applied to DevOps.

    Judith Hurwitz: And so, Brian, I think that you are 100 percent right. And that’s when commercialization really happens, is when we have those standards and when everybody agrees to use those foundational services. I think the challenges that we face are, how to get the commercial vendors who don’t necessarily want to –

    They want to give lip service to these standards, but they really want you to only use their version of the “standard” so that customers never leave. So, it’s that stickiness factor that they are looking for. So, I think it’s it’s a hard journey.

    Brian Dawson: Yeah. Well, I think, Judith, you actually nailed the reason the Linux Foundation exists, frankly. And if we look at Kubernetes as a model, it was hard. Press Google could have easily said, “We’re not going to hand this over to the Linux Foundation. We’re gonna dominate this space. We are going to be the modern cloud OS.”

    But they understood for it to gain traction, for it to grow and truly offer benefit industrywide, they had to hand it over to the Linux foundation, a grow manage. They had to bring in Microsoft. They had to bring in Amazon to play in that space. And I see LFAI & Data serving that same role, what I tend to call in some language unlocking innovation, or as our tagline is, decentralized innovation.

    And I would beg to say that that is, we didn’t call that out directly, but one of the challenges that we’re seeing in the AI and data space. If there’s not short-term monetization, we’re not gonna do it. And if there is, then we wanna own it. So can we create a impartial playing field for everybody to come in and innovate together, and then build commercial solutions off of that?

    Judith Hurwitz: Yeah, it’s the challenge. It really is. And I think Kubernetes is a great example. I think data in some ways is more complicated because data’s really the crunch –

    Brian Dawson: Yeah. Well, I know everybody wants to own it. Everybody wants to own the data.

    Alan Shimel: No doubt about it.

    Mitchell Ashley: There’s discussion about AI needs to become an engineering discipline as opposed to this sort of edge specialty. Doesn’t mean that necessarily is gonna be applying everywhere, but I think how we apply DevOps or, I’m sorry, AI to certain kinds of problems, it seems to me the two most ripe areas are that highly complex environments.

    So it’s more and more infrastructure, it’s automated, more things are gone to digital. How do you manage the infrastructure or manage the triage problem-solving? And then of course, I think that the other areas, people will continue to find niche areas where AI can be applied to gain competitive advantage in a certain domain or space.

    And it seems like that’s the trajectory we’re on for quite a while. I don’t think it’s ever gonna be AI takes over everything, and it’s the thing that replaces DevOps or whatever. But it seems to me kinda a tool, but – Sorry, go ahead, Judith.

    Judith Hurwitz: No, what I was gonna say is very interesting. If you look at healthcare, for example, if you look at the spectrum firm, I can automate certain DevOps functions that are repeatable, where I can identify patterns. And as I collect more data, I can automate more things.

    You have something like healthcare, which in terms of orders of magnitude of complexity are just huge. And what I’m seeing is a lot of the vendors who think that I’m going to tackle healthcare with AI and we’re gonna own the industry, a lot of them are getting out of the business because right now, it’s just too hard and it will be too hard for quite a while.

    Mitchell Ashley: It’s true of the medical industry in many ways. It’s a hard nut to crack.

    Brian Dawson: Yeah. If I may start to dream a little and chime in based on something you said Mitch, I do see that in the near term, we are at a point where we talk about modern software development and delivery, the rapid pace which we’re delivering change, we’re building on inventions and progress made over decades. Library reuse is really heavy.

    And we are getting to a point where as we build more complex systems, we have to figure out how can we outsource sort of maintenance and management for one of a better word to use it kind of grossly to solutions like ML? How can ML help us continue to improve, grow and build on what we’ve done, but manage the scale and complexity?

    And I think we’ll move from some standardization, foundational blocks will apply more ML and both the operations and development space to manage that complexity, to maintain stability. But then I also eventually see the next stage being now, how do we apply ML and surface it to a developer at the time of a commit?

    Here’s what the expected outcome of this changes to help provide guardrails to the developer, the end state, or I wouldn’t even say the end state. I think we’d agree existentially there’s never an end state here. That we’re at a point where we don’t even have explicit CICD pipelines. We can commit code to a repository, the language can be inferred.

    We can give queues or signals to infer where we want it deployed, how we want it deployed. And I actually see ML and to an extent, layers of AI, just figuring out what to do with code at rest. So if you flash forward in 10 years, and we’re truly applying these to technologies at scale in the cloud, I just change a line of code.

    That change automatically is delivered to production. And AI/ML is helping us do that. So that’s sort of my dream. I don’t hit compile, I don’t hit build, I don’t build out stages and workflows, I just change code, and that’s running in a system somewhere.

    Judith Hurwitz: So, don’t you also want to have the ability to say before you say, you look at what’s happening, and don’t you wanna sort of at least at this stage say, “Are you sure you wanna do that? Are you sure you wanna delete that file?”

    So until we get to the point, because you have to be able to trust that the system is smart enough to understand if I make this change, what cause will that unleash? Because we’re not dealing with the perfect world in DevOps by a long shot.

    Brian Dawson: Yeah, I do want that. I want protection. I want it to know where the vulnerabilities are and warn me. And I do think, Judith, that would be a prerequisite into this deploying code at rest, or truly applying dev AI.

    Alan Shimel: Fair enough. Hey guys, we are just about out of time. I think this was a great discussion. If there’s one thing I could take out of it though, is that the crystal ball remains cloudy in terms of how this is all gonna play out and we’re gonna have to wait for it to kind of come into focus as we – But it certainly will be an important part of software development, of operations going forward. And it’s gonna be an increasingly important part.

    Brian Dawson: [crosstalk]

    Mitchell Ashley: I think the crystal ball is in Magic 8-Ball, ask again later.

    Alan Shimel: That’s a good one, Mitch. But for now, we’re gonna call an end to this episode of DevOps Unbound. Again, thanks very much to Tricentis for their sponsorship. Thank you so much, Judith. Thank you so much, Brian, for appearing here and hope to see you on a future episode. Mitch, as always, great job, riding shotgun with me.

    Mitchell Ashley: Good to partner with you. Absolutely.

    Alan Shimel: Absolutely. But this is it for this episode of DevOps Unbound. This is Alan Shimel, have a great day, and we hope to see you soon on another DevOps Unbound, as well as don’t forget, every month we do a live round table open to you, our audience, with questions. So stay tuned for that as well. Take care, everyone. Bye-bye.

    [End of Audio]

  • Apple Allows 50% Fee Rise | @ElonMusk Fans: 70% Fake | Microsoft Salaries up by 100%?

    Apple Allows 50% Fee Rise | @ElonMusk Fans: 70% Fake | Microsoft Salaries up by 100%?

    Welcome to The Long View—where we peruse the news of the week and strip it to the essentials. Let’s work out what really matters.

    This week: Apple lets devs increase subscriptions “without consent,” Twitter has more than 19% spam accounts, and we dig into the claim that Microsoft is about to double salaries. (more…)

  • Top 3 Requirements for Next-Gen ML Tools

    Top 3 Requirements for Next-Gen ML Tools

    As the machine learning market matures, new tools are evolving that better match data science and machine learning teams’ needs. Vendors, both open source and private, have been quick to introduce new products that better meet requirements making it easier and faster to develop models and enable collaboration.

    These new offerings range from cloud-based platforms that make it easy to build and deploy models to specialized software that can speed up the training process. As an example, the existing field of ML experiment tracking is a multi-billion dollar market with a crowded amount of vendors: Comet ML, Weights & Biases, Neptune AI, Databricks’ MLflow and more. These tools will need to evolve to move beyond meeting only the basic requirements of ML teams like experiment tracking, visualizations and model training to address collaboration and technical requirements so these teams can get models to market faster.

    The types of teams that are driving modern requirements for data science are the bet-the-farm types, where it’s all about finding new technology and solutions to drive efficiencies and get models to market faster for their organizations to maintain a competitive edge. Going forward, we’re seeing the following as requirements for ML teams as they evaluate tools to add to their MLOps stack.

     An Open, Modular Approach

    Data scientists and ML engineers use multiple tools to do their jobs, from Jupyter notebooks for model coding to DVC for experiment and data versioning, to automation tools like AirFlow to model visualization tools. Solutions will need to be open to integration with other tools instead of a walled garden that many ML tools, like those offered by cloud vendors (e.g. Amazon SageMaker, etc.) are now.

    Familiar Developer Experience

    ML developers and engineers are familiar with standard DevOps tools and workflows like GitOps (think GitHub or GitLab) and live in CLI-oriented solutions. ML tools should integrate tightly with these existing tools instead of complicating the developer experience with a separate ecosystem. And as a part of the developer experience, workflow is also important. Data scientists can run hundreds of experiments using their preferred CLI-based solutions but, often, all those experiments are streamed and posted into existing ML tools. A more optimal approach would be to offer data scientists the experience of running experiments locally and letting them commit only the most important experiments for further discussion and collaboration so the team isn’t overwhelmed.

    Strong Connection to Source Code

    This approach makes a stronger foundation for something like tracking ML experiments. As ML teams race to get models to market and continually improve them, being in lockstep with DevOps teams to deploy those ML models into production is critical. An example would be a developer tracking five experiments with changing hyperparameters. For these experiments, they may have a link to a Git repo for a particular version of source code. But what if the source code changes or data set changes are not committed or reported in their existing ML solution? Then, the connection is lost and existing metrics would be based on a previous version of source code or data that’s no longer accurate. That lost connection slows down time-to-market and makes it harder to optimize models.

    As seen with the above examples, ML teams juggle myriad tools and workflows as they develop models for various applications and services. The requirements discussed will help these teams to collaborate better and build models faster in an increasingly heterogeneous MLOps landscape.

  • What is MLOps? DataOps? And Why do They Matter?

    What is MLOps? DataOps? And Why do They Matter?

    Let’s look at three distinct disciplines—DevOps, MLOps and DataOps. In 2011, Marc Andreessen famously proclaimed that software was “eating the world.” A little more than a decade later, it’s all but impossible to argue with his premonition as software has embedded itself into virtually every industry. 

    But now another shift is underway and AI is gobbling up software as we know it. Our emails can finish our sentences for us, connected cars are helping us stay safer on the road and predictive technology is helping minimize supply chain disruptions caused by COVID-19. In the near future, almost every piece of software we interact with will have intelligence built-in, and as AI gains acceptance and new use cases are rolled out, applications will only continue to become more intelligent. The widespread move toward ML-enabled software has the potential to be as transformative and far-reaching as what Andreessen predicted. 

    But innovation is rarely easy or simple—and intelligent applications are no exception. While conventional software involves one thing—code—and not much else, intelligent software relies on a complex relationship between three interconnected variables or legs of a three-legged stool.

    A Three-Legged Stool: DevOps, MLOps and DataOps

    Model:  The one or more AI/ML models (e.g., linear regressions, rules, deep neural networks, etc.) trained to recognize patterns in data and make decisions are what makes an application “intelligent.” 

    Data: Most often, a model is trained on historical data and programmed to emulate it. As a result, an application’s behavior depends heavily on the data—the raw input data, labels and features, as well as the new data to which the model is applied and ground-truth for this data.

    Code: Code is the language an application uses to function. When used in the context of intelligent applications, the code pillar might refer to business logic, calls to models, receiving outputs, decision making and calls to other data systems. 

    A Real-World Example

    To see how these pillars play out in real life, consider the example of a perception system for a self-driving car. Such systems use data (provided by offline sources and real-time cameras and sensors), models and code to determine myriad ever-changing variables (for example, if there are obstacles on the road, what lane markings look like, how fast a car can safely travel, etc.) While the following illustration is greatly simplified, it outlines the basic steps needed to build and deploy such a system and indicates which pillar each step belongs to. 

    Regardless of application type, the legs that intelligent applications stand on are always the same: Model, data and code. And if you try to remove any one of them, the application will topple over.

    With the three distinct pillars that make up intelligent applications, three distinct disciplines have emerged to keep each individual pillar functioning as efficiently as possible: DataOps, MLOps and DevOps. The rest of this post examines the details that make each discipline distinct and necessary to the model life cycle.

    DevOps

    Once upon a time, companies used waterfall processes to develop software. The process often moved so slowly that products didn’t realize their potential before they died or were replaced. In 2000, Agile was introduced, allowing developers to work quickly and iteratively in tight sprints where they were constantly delivering. By 2009, Flickr, for example, was the envy of the industry, routinely performing 10+ deploys per day. Companies seeking to increase their own productivity combined software development with established operations principles—and the term “DevOps” was coined. 

    As the familiar loop graphic below illustrates, the DevOps cycle goes from the planning stages to the creation, testing/verification, packaging, release, configuration, monitoring and then starts over again with planning. Over the past two decades, a host of unique tools have emerged to support individual steps in the DevOps life cycle—helping developers ensure quality by building in continuous integration and continuous testing from the start and helping speed up time-to-market with workflow automation. The DevOps ecosystem is robust, well-thought-out and always iteratively improving.

    MLOps

    MLOps helps teams move more quickly and efficiently through the model life cycle—a loop that involves six distinct steps, each of which contains unique considerations. 

    Since MLOps is related to DevOps, businesses sometimes try to adapt the processes used to write conventional software to execute the unfamiliar task of machine learning operationalization. But the approach doesn’t work, as the two disciplines are actually quite different.

    MLOps primarily deals with models, not code—and models are quite different from code. For example, model training and testing look very different from code building and testing. (What does “test case” even mean for a model?) And monitoring model performance is completely different than monitoring traditional software. (DevOps looks at things like CPU utilization, latency and throughput, while MLOps looks at model quality, drift and data quality.)

    Furthermore, the people that MLOps engineers are building for are quite different than their DevOps counterparts. As one might expect, the primary users of models are data scientists while DevOps generally serves software developers. Most often, MLOps engineers come from a background in ML and data science or software engineers who picked up ML. 

    As with DevOps, no single all-inclusive MLOps platform exists. Rather, different tools support different individual steps in the model life cycle and provide some amount of workflow automation to speed up processes. Different tools support specific pieces of the ML life cycle—but the ecosystem is still somewhat fragmented. Ideally, MLOps will evolve to offer more DevOps-like tools that work well together and offer more automation. For companies looking to build an MLOps platform, build a suite of supporting tools with speed, safety and automation in mind.

    DataOps

    We’ve seen very well-defined life cycles for the other two pillars discussed above; however, when it comes to DataOps, the life cycle is still being defined. But the overall objective of DataOps should sound familiar: DataOps involves processes and tools to ship high-quality data frequently—which requires a combination of data engineering, data quality, data security and data integration. 

    As with DevOps and MLOps, what the DataOps platform or DataOps engineer is doing is supporting the tools that help perform one of these activities and help build workflows. For example, a workflow may take data from a database, apply some transformation to it and then make it available for a business intelligence (BI) tool. That’s the kind of automation that is involved in DataOps. 

    As you might imagine, the skillset and the tools required for DataOps are quite different from the ones needed in MLOps and DevOps—so expecting a DevOps engineer to also be an expert in DataOps seems a bit unrealistic. While MLOps requires a much tighter integration with data than DevOps historically has required, MLOps is not the same thing as DataOps. More so than the other pillars that support intelligent applications, DataOps tends to be quite separate and distinct. 

    Conclusion

    From perception systems to enterprise-scale fintech software, every intelligent application depends on three distinct disciplines: DevOps, MLOps and DataOps. And therein lies the complexity of developing intelligent products. Each of these fields is distinct, dealing with a different set of questions and objectives within the ML life cycle and requiring different kinds of people and tools. However, they are all fundamentally united by a common goal: Optimizing quality and speed of iteration of the ML life cycle. The tools, practices and organizations that will rise to the top are the ones that will enable more seamless collaboration between different ML teams, helping them move through the ML life cycle quickly and efficiently. 

  • Unreliable Server Scare | Information Batteries | ARM IPO PDQ

    Unreliable Server Scare | Information Batteries | ARM IPO PDQ

    Welcome to The Long View—where we peruse the news of the week and strip it to the essentials. Let’s work out what really matters.

    This week: We worry about chips failing randomly, we ponder a new way of thinking about workload shifting, and we grok Arm’s IPO.

    1. Smaller Features Lead to Random Failures

    First up this week: Could cosmic rays be causing increasing numbers of errors in cloud servers? As chips get faster and more densely packed, the worry is that undetected bit-flips could cause serious problems.

    Analysis: How can you mitigate the problem?

    As workloads grow in number and complexity, this is becoming a real issue—not merely theoretical. (See also: DevSecOps worries such as Rowhammer.) Think about where the critical data are and how to detect random errors. Even consider running important workflows in lockstep pairs.

    John Markoff: Tiny Chips, Big Headaches

    Imagine for a moment that the millions of computer chips inside … the largest data centers in the world had rare, almost undetectable flaws. … As the tiny switches in computer chips have shrunk to the width of a few atoms, the reliability of chips has become another worry for the people who run the biggest networks. … There is growing evidence that the problem is worsening with each new generation of chips.
    …
    Both Facebook and Google have published studies describing computer hardware failures whose causes have not been easy to identify. … Facebook researchers noted that some processors would pass manufacturers’ tests but then began exhibiting failures when they were in the field. … Smaller transistors, three-dimensional chips and new designs that create errors only in certain cases all contributed to the problem … according to Google.
    …
    Intel executives said they were familiar with the Google and Facebook research papers and were working with both companies to develop new methods for detecting and correcting hardware errors. [But] computer engineers are divided over how to respond to the challenge.


    Have we hit a wall? So asks Thutmose V:

    For more than 50 years, semiconductor manufacturers have been trying to keep up with Moore’s Law, which … has effectively served as the target for semiconductor engineers to push them forward. … The size of some parts of the transistors is now 10 nanometers or less. 10 nanometers is about equal to the width of 100 silicon atoms—100 atoms is not very many.
    …
    When things are that small, does quantum randomness start to be noticeable? Normally if the odds of something failing is a trillion to one, you don’t need to worry. But if you have 10 billion transistors operating at a clock speed of 3 billion times a second, a one in a trillion chance of failure is nowhere near good enough. … It is a miracle these things work as well as they do.


    We’ve known about the problem for years, but only now is it becoming reality, says dsgrntlxmply:

    We observed certain network failures to be caused by corruption of an on-chip routing table memory in a SoC that lacked parity protection for these tables. The failure rate was low, but was higher in units that were deployed at higher altitudes. … The problem was eventually mitigated by software table sweeps, and in newer SoC generations, by parity in critical tables.

    These Single Event Upsets are said to be caused by muons generated by high energy cosmic ray interactions in the upper atmosphere. Third semester undergrad physics.

    2. Mind-Bending Speculative-Compute Research

    Many of us are familiar with delaying a workload until a time when servers are idle—or when energy is cheap. But some fascinating research asks: What if we could predict a future workload and perform it in advance?

    Analysis: “Information batteries” can be cheaper than Li-ion

    The researchers argue that pre-caching even parts of a workload’s output when energy costs are low—or even negative—can save far more money than traditional grid-scale storage such as batteries or pumped hydro. For example, speculatively transcoding video versus generating it on the fly, or predicting future machine learning training workloads.

    Caitlin Dawson: A New Way to Store Sustainable Energy

    Renewable energy has an intermittency problem. … What if surplus renewable energy could be stored as computation? … That’s the thinking behind “information batteries,” a new system proposed by … Barath Raghavan, an assistant professor in computer science at the USC Viterbi School of Engineering … and Jennifer Switzer, a Ph.D. student from UC San Diego.
    …
    When renewable energy is available in excess, it is used to speculatively perform computations. … The stored computed results can then be used later when green energy is less plentiful. … For instance, every day, YouTube data centers transcode more than 700,000 hours of videos to different resolutions. Many of these computations are predictable and can be performed at a time when there is excess green energy.
    …
    [But] computations that are completed in advance do not need to match exactly with the computations completed at a later time. … The challenge, said the researchers, is determining what computation to perform, where and when, and how … to efficiently retrieve the results later. … It is only possible in some workloads and in some contexts. But Raghavan believes with improved prediction and integration into large systems, the technology points towards a promising future alternative.


    This isn’t just batch computing with a fancy name. cognizant_ape explains:

    On AWS we do a lot of batch processing when the data centers are low on traffic. This is just the standard SPOT instances which you can trigger once the cost goes below a certain amount and then end when it goes about a certain amount.

    The big issue is being effective with I/O and setup time. The article seems to be getting ambitious to the point of a distributed compiler with a flexible optimization metric.


    Cut to the chase. It hinges on prediction, says bradley13:

    Predicting loads is easy, but predicting the computations that will make up those loads? Not so much. … Imagine: Most employees start work in an hour — let’s predict which database queries they will generate. Your success rate better be high.

    3. Confirmed: Arm will IPO, Not be Sold to Nvidia

    SoftBank and Nvidia have bowed to the inevitable. Arm, designers of ubiquitous chip cores, will IPO sometime in the next 12 months.

    Analysis: The only sane choice, but ecosystem risk remains

    As I’ve said before, it seemed everyone was against the idea of Nvidia owning Arm. It put the kibosh on Arm’s open licensing stance, dooming the ecosystem to scraps from the table as Nvidia grabbed first dibs on everything. Now regulators must turn their attention to the very real risk that Nvidia will still acquire Arm anyway—via the stock market.

    Richard Waters, Arash Massoudi, James Fontanella-Khan and Antoni Slodkowski: $66bn sale of chip group Arm to Nvidia collapses

    The deal, which would have been the largest ever in the chip sector, was set to give … Nvidia control of a company that makes technology at the heart of most of the world’s mobile devices. A handful of Big Tech companies that rely on Arm’s chip designs, including Qualcomm and Microsoft, had objected to the purchase [arguing] that Nvidia would get an unfair advantage by having first rights to Arm’s technology.
    …
    Jensen Huang, Nvidia’s chief executive, hoped to use Arm’s processor designs to cement his company’s growing role in data centres. … Nvidia abandoned its pursuit of Arm at a board meeting on Monday, said a person familiar with the discussion.
    …
    The deal’s failure brought an immediate management shake-up at Arm, with chief executive Simon Segars replaced by Rene Haas, head of the company’s intellectual property unit. … In the UK, where politicians view Arm as a strategic national asset, attention is set to shift to whether the company will be listed in London [but] Haas said … no decisions had been made on where Arm would be listed.


    Reminding us why this matters to DevOps, here’s Timothy Prickett Morgan:

    Now there is no question that Arm Holdings will be going public. … The plan is to do so before the end of [its] fiscal year in March 2023. … Arm Holdings will be returning to its IP licensing roots, having spun off some uncore business back into SoftBank, and seeking out new markets as it has been able to do with hyperscalers [and] cloud builders.
    …
    When Intel was worried about AMD Epyc chips, what it really needed to worry about was boosting price/performance to keep Arm CPUs out of the hyperscaler and cloud builder datacenters. [But AWS] has Graviton Arm server chips on a pretty regular schedule, and you can bet Masayoshi’s last yen that Graviton4 will debut at re:Invent 2022 this year.


    But what happens next? Pun it up, HereAndGone:

    If Softbank is going to take ARM public, Nvidia will find it much cheaper simply to acquire a controlling interest. … Looks like a duck, walks like a duck, but cheaper than a duck—while ducking regulatory issues. This plan is not quacked up to solve what it pretends to solve and will likely prove most fowl.

    The Moral of the Story: Fall seven times and stand up the eighth


    You have been reading The Long View by Richi Jennings. You can contact him at @RiCHi or tlv@richi.uk.

    Image: Nicole Geri (via Unsplash; leveled and cropped)