Category: Continuous Testing

Continuous testing is the process of executing automated tests as part of the software delivery pipeline to obtain immediate feedback on the business risks associated with a software release candidate.

  • New Relic Adds Module to Surface Code Metrics

    New Relic Adds Module to Surface Code Metrics

    New Relic today added a free CodeStream module to its observability platform to give developers access to metrics and telemetry data that will enable them to write higher-quality code faster.

    Peter Pezaris, group vice president and general manager of strategy and experience for New Relic, said this capability is part of an effort to shift observability left toward developers to optimize code as it is written and after it is deployed.

    CodeStream surfaces metrics and telemetry data within the integrated development environment (IDE) developers use to write software. Telemetry metrics are displayed as a line of text above the code or in the debugging tool used to troubleshoot an application. Alternatively, developers can view metrics via an observability panel at the service level. They can also view performance metrics such as error rates and response times for related services.

    That approach eliminates the need to wait for IT operations teams to surface issues that usually don’t manifest until long after the code in question was originally developed, noted Pezaris.

    The overall goal is to provide development teams with frictionless access to observability data at every stage of the software development life cycle, he added. That analysis capability will result in faster mean-time-to-detection (MTTD), mean-time-to-resolution (MTTR) and, ultimately, shorter development cycles, said Pezaris.

    Organizations should also be able to reduce technical debt faster by making it easier to identify issues that, for example, adversely impact application performance, he noted. That activity can be especially challenging because developers often inherit code written by someone else that often lacks clear documentation. CodeStream makes it simpler to onboard a developer to any application modernization initiative that has been launched to reduce technical debt, said Pezaris.

    A recent New Relic survey found only 27% of respondents have achieved full-stack observability, and only 5% claimed they have a mature observability practice in place. A third (33%) of respondents also said they still primarily detect outages manually or based on complaints, the survey found.

    On the plus side, the survey also found nearly three-quarters of respondents said C-suite executives in their organization are advocates of observability, and more than three-quarters of respondents (78%) saw observability as a key enabler for achieving core business goals. However, more than half (52%) of respondents said they experienced high-business-impact outages once per week or more, and 29% said they take more than an hour to resolve those outages.

    A full 82% of respondents said they use four or more tools to monitor the health of their systems. Only 7% said their telemetry data is entirely unified and only 13% said the visualization or dashboarding of that data is entirely unified. Nearly half of the respondents (47%) said they would prefer to have a single, consolidated observability platform, but only 2% said they currently used one tool for observability.

    Regardless of the approach to observability, it is clear that development and IT operations teams need to be looking at the same metrics and telemetry data at the same time to collaborate effectively.

  • DevSecOps Provides a Modern Security Model for Modernization

    DevSecOps Provides a Modern Security Model for Modernization

    Developers and security experts are now tasked with bolstering, extending and adjusting cloud and Kubernetes security to protect against cyberattacks that are ever more complex, volatile, and frequent. To foil attacks and create a secure foundation for applications and infrastructure from the beginning, DevSecOps (development, security and operations) has become the trending development and operations practice. In the DevSecOps model, security becomes a shared responsibility. Thus, DevSecOps requires a mindset shift to enable collaboration between development, security and operations teams. This shift, commonly referred to as shift left, involves culture, processes and tools. With DevSecOps, whether it’s testing for security vulnerabilities or building security services that can be used directly for business purposes, the goal for all participants is to build security into applications from the start and throughout the continuous integration and continuous delivery (CI/CD) workflow of DevOps.

    DevSecOps to the Rescue

    In the past, security considerations and practices were often introduced at the end of the development cycle by a separate security team and tested by a separate (QA) team. This was manageable when software updates were released every few months or even years. But in today’s landscape, consumer habits and expectations are shaped by smartphones and digital commerce, which has fueled the demand for software services that are real-time and available 24/7. This requires modern enterprises to find ways to improve the efficiency of application developments, releases, and updates.

    Because of these factors, enterprises realize that security can no longer be a small part of an afterthought in the production environment. The purpose of DevSecOps is to ensure that through collaboration, security can be carried through to every point in the development cycle. This enables teams and enterprises to deliver secure, high-quality applications in a more efficient manner without extensive security checks and fixes occurring during post-production.

    To meet these more extreme demands, cloud computing, containers, and microservices have made it possible to accelerate the development and delivery of software releases. Developers adopting agile and DevOps practices can reduce software development cycles to days and weeks, thus meeting the diverse needs of enterprises and users.

    This fast-paced development and upgrade frequency has created new security concerns and the need for companies to be more agile in responding to security issues. DevSecOps has been introduced into the DevOps framework to meet these needs by make security a shared responsibility. Today, continuous testing and integration, including security scanning of pipelines, is becoming the norm.

    DevSecOps Business Benefits

    In principle, DevSecOps shares the DevOps ideal of multiple teams working together to improve team efficiency and achieve secure, continuous software delivery.

    In addition to the security benefits DevSecOps provides, there are significant business advantages to be gained, including:
    ● Efficiency–Under the DevSecOps practice, security is integrated into all periods of development to help all teams be more agile in responding to security risks, eliminating the need for teams to spend a lot of time tweaking and fixing during the production cycle.
    ● Cost reduction–By discovering security vulnerabilities before they enter production, organizations and teams can significantly reduce the time and labor costs of fixing them.
    ● Ensure compliance–DevSecOps can ensure compliance with industry-standard regulations, such as the General Data Protection Regulation (GDPR). DevSecOps gives teams a holistic overview of these measures that makes compliance easier.
    ● Establishes collaborative culture–Integrating security practices into DevOps enhances the value of DevOps and improves the overall security posture as a culture of shared responsibility. When everyone is involved in the process, it increases their awareness of security fundamentals and best practices and provides a sense of ownership in the results.

    Meeting New DevOps Challenges

    Kubernetes offers many advantages but also poses unique security challenges that can be difficult to address for organizations lacking in Kubernetes talent and experience. This is why organizations will increasingly see the need to reevaluate their security practices and prioritize a more advanced security-focused culture.

    Because DevSecOps requires security to be addressed throughout all development stages, it requires developers to have security expertise while coding and operating. We are currently seeing a growing skills gap in the DevOps industry, and developers are feeling burned out. Security training is a way for teams adopting DevSecOps to acquire additional knowledge.

    Deploying Kubernetes platforms with security built in by default also will be recognized as a means to reduce the burden of security on IT teams. This will reduce the pressure and burnout on all sides.

    Platform Engineering Provides Additional Help

    With the rapid development of Kubernetes and cloud-native applications, organizations are realizing the inadequacies of their IT teams to leverage DevOps practices. Just as platform engineering eases the burden of DevOps by providing an Internal Development Platform (IDP) that serves as a “golden path” for developers, an IDP can simplify the practice of DevSecOps.

    We’ve seen that a DevOps workload is difficult to practice in small and medium-sized enterprises, as well as in large enterprises that lack sufficient talent. The gradually accumulated cognitive load ultimately leads to a less agile and efficient collaboration between teams. Given these issues, more organizations will adopt platform engineering as a better alternative.

  • Linux Tweak Brings Big Speedup ¦ DCs in SPAAACE (Redux) ¦ Atlassian Fires 500

    Linux Tweak Brings Big Speedup ¦ DCs in SPAAACE (Redux) ¦ Atlassian Fires 500

    Welcome to The Long View—where we peruse the news of the week and strip it to the essentials. Let’s work out what really matters.

    This week: Intel optimizes Linux multithreaded networking, data centers in space (again), and more DevOps toolmaker layoffs. (more…)

  • Checkly Adds Monitoring-as-Code to Testing Platform

    Checkly Adds Monitoring-as-Code to Testing Platform

    Checkly this week added a monitoring-as-code (MaC) workflow capability to its software-as-a-service (SaaS) platform that is used to create synthetic tests for applications.

    The Checkly platform spins up browsers to test and can now monitor how applications are performing at both the user interface and application programming interface (API) level. That capability, currently available in beta, is enabled via a command line interface (CLI) for accessing an instance of TypeScript to invoke MaC workflows.

    Checkly CEO Hannes Lenke said that integrated approach enables application development teams to observe applications in as little as 60 seconds without instrumenting them with agent software.

    As part of that effort, Checkly also made generally available Playwright Test, a testing platform for web applications that spins up modern browsers on demand in addition to updating its public dashboards and adding an analytics API.

    Finally, the Checkly SaaS platform has passed a SOC 2 Type II security audit.

    Lenke said the overall goal is to make it simpler for developers to discover and remediate issues as they assume more responsibility for the ongoing management of applications.

    It’s not clear whether developers are taking over responsibility for testing and monitoring applications, but as more responsibility for applications shifts left, organizations are looking to give developers the tools they will require to succeed. That doesn’t mean IT operations teams won’t need to continue to monitor IT environments, but developers do need integrated tools that give them enough meaningful context to address them as they write code. Synthetic testing tools also enable developers to address many routine issues long before they might be discovered by a dedicated testing team that should be focused on ensuring the overall quality of the user experience.

    Regardless of the approach to testing and monitoring, Lenke noted that the silos that make up the software development life cycle are becoming more tightly integrated in modern application development environments. The primary objective is to make it possible for organizations to accelerate application development without compromising quality, he added.

    Of course, the more testing that occurs early in the application development process, the fewer issues a DevOps team should encounter after an application has been deployed in a production environment. The issue is that, historically, much of that testing occurs late in the application development process. Each time a project falls behind schedule, there is naturally a greater temptation to reduce the amount of time spent on testing to ensure the on-time delivery of an application. Providing developers with tools for testing and monitoring applications as they are being built should ensure that the proper level of testing is occurring before an application is deployed.

    There will, however, never be a perfect application. Over time, DevOps teams will discover that every application is impacted by drift to varying degrees as updates are made and the underlying infrastructure is upgraded. The most critical issue is reducing the number of potential issues overworked DevOps teams may need to address later on by ensuring the code being deployed is of the highest quality possible.

  • WTH? WFH is 6× Pre-Covid ¦ Plus: Agile Sucks (Redux)

    WTH? WFH is 6× Pre-Covid ¦ Plus: Agile Sucks (Redux)

    Welcome to The Long View—where we peruse the news of the week and strip it to the essentials. Let’s work out what really matters.

    This week: Working from home is here to stay, and Agile is still a failure. (more…)

  • Tabnine Extends Generative AI Platform Reach to Testing

    Tabnine Extends Generative AI Platform Reach to Testing

    Tabnine today announced it extended its generative artificial intelligence (AI) platform that enables developers to write code faster to now include the ability to automatically generate tests for that code.

    Available in beta, this capability promises to make it easier for developers to test code at the time it is being developed. That approach should make it simpler for developers to address issues without losing any context.

    Brandon Jung, vice president of ecosystems and business development at Tabnine, said this latest addition to the Tabnine platform is part of a larger effort to extend the reach of the Tabnine generative AI platform across the entire software development life cycle. The testing capability is expected to be generally available in the third quarter, and Tabnine expects to also employ generative AI to automate a much wider range of tasks across the software development life cycle within the next 12 months.

    The Tabnine platform supports multiple programming languages including Python, Java and JavaScript and is designed to be integrated with integrated development environments (IDEs) such as Visual Studio Code and Jetbrains. The company has also previously integrated its code completion tool with the GitLab continuous integration/continuous delivery (CI/CD) platform. The overall goal is to make it easier for developers to automatically write code based on custom models using approved source code hosted in a secure private repository, said Jung.

    Generative AI creates a large language model that assesses the probability of what the next line of code will be based on what has preceded it. Tabnine is now extending the model it created to also generate tests for the code its platform helped a developer write. It’s not clear what impact this ability to automate test creation will have on the need for dedicated testing teams, but it is about to become considerably easier to shift more responsibility for testing further left toward developers.

    It’s not likely DevOps teams will be replaced as generative AI is extended across workflows, but the overall size of those teams might shrink, noted Jung. DevOps teams will soon be able to do far more with fewer people, he noted. At the same time, the barrier to DevOps adoption will also fall as AI platforms make it simpler for more organizations to embrace DevOps best practices, added Jung.

    The challenge will be making sure the data collected to train AI models is of a high enough quality to ensure the desired outcome, noted Jung.

    One way or another, it’s now only a matter of time before generative AI capabilities are applied more broadly. Platforms such as ChatGPT developed by OpenAI are only the tip of the iceberg that impacts almost every manual process, including software development and deployment. The issue will be determining at what rate those innovations will become practical to employ. In the meantime, it is already apparent generative AI platforms are having a significant impact on the rate at which code can be developed. Inevitably, that means the amount of code moving through DevOps pipelines at any one time is about to increase significantly.

  • Tricentis Extends Mobile Testing Reach

    Tricentis Extends Mobile Testing Reach

    Tricentis has added the ability to test native iOS and Android applications to its Testim software-as-a-service (SaaS) platform.

    Previously, the Testim platform was designed to test web applications, but is now being extended as organizations build and deploy more native mobile applications. Capabilities of the Testim Mobile platform include codeless authoring of mobile tests to eliminate the need to write and maintain Appium, Espresso or XCUITest scripts and the ability to set up, configure and manage mobile devices in minutes using the Tricentis Mobile Agent.

    Additionally, applications can be uploaded to the cloud to enable collaboration across testing teams as they opt to run tests on real or virtual devices. DevOps team can also customize tests with JavaScript, branch tests, and run tests from a scheduler or continuous integration (CI) build.

    Tricentis CTO Mav Turner said Tricentis is moving to implement a consistent set of mobile testing services across its entire portfolio of testing platforms now that mobile applications are equal, first-class citizens alongside web applications. The primary driver of that shift has been the COVID-19 pandemic, which saw more organizations relying on external-facing mobile applications to engage customers, partners and employees working remotely, he added.

    Testing those applications is more critical than ever because the cost of failure—in terms of impact to the business—is now much higher as more organizations depend on software to drive business processes, noted Turner.

    The goal is to make it simpler to test these applications as they are being developed to ensure quality, he added. The challenge is that many developers don’t have a software engineering mindset. Hence, they need to rely more on testing platforms infused with artificial intelligence to discover issues relating to, for example, scalability and security, said Turner.

    Most organizations are trying to shift more responsibility for testing toward developers as part of an effort to remediate issues earlier in the software development life cycle. It’s more challenging for developers to remediate issues after an application has been developed simply because they have lost context. Testing platforms infused with AI capabilities make it easier for developers to identify issues at the time they are writing code. In the future, Tricentis expects to take advantage of generative AI platforms to make it easier to author tests using a set of best practices that can be easily reused, noted Turner.

    That approach doesn’t necessarily eliminate the need for a dedicated quality assessment review, but it does allow testing teams to focus more of their time and effort on testing the application experience versus identifying lower-level errors that many developers routinely make.

    Over time, the quality of applications should steadily improve as many of the low-level mistakes that developers make are eliminated. That should improve everything from application security to the user experience as applications are continuously updated. In the meantime, organizations that embed testing within DevOps workflows are steadily reducing the number of issues that need to be addressed just before applications are supposed to be deployed.

  • How Will Low-Code and AI Coexist?

    How Will Low-Code and AI Coexist?

    Over the last few years, low-code solutions have become important to help organizations widen programming accessibility to a broader base of users and help fill development gaps. And these technologies are still gathering steam—Gartner estimated the market for low-code development technologies would grow to a total of $26.9 billion in 2023 and that 65% of applications would be developed using low-code by 2024.

    However, AI-based code generation is set to further level the playing field with advanced auto-completion and AI coding assistants. Take GitHub’s Copilot, a helpful AI pair programmer, or OpenAI’s ChatGPT, which can generate complex functions given natural language prompts. Other AI tools trained on UIs, like Galileo, can derive frontends for any project in a matter of seconds. So, are low-code development platforms still relevant in this new paradigm of AI-driven software development? If so, how will low-code solutions and AI coexist?

    Well, generic algorithms can only go so far in creating truly unique and innovative experiences that solve real-world business cases. And since most programmers don’t have the skills to train machine learning or deep learning models from their own datasets, this is one area where low-code solutions could lower the barrier, enabling users to tag unstructured data, generate models, run simulations and share reusable AI across departments.

    I recently chatted with Johanna Pingel, MathWorks AI product marketing manager, to discuss the role of AI in low-code programming. According to Pingel, low-code solutions can lower the barrier to AI functionality, both in quickly integrating AI-driven functionality into applications and seamlessly generating new AI algorithms based on unique datasets. Below, we’ll consider some additional use cases for AI and how low-code platforms may work alongside it.

    Use Cases For AI

    First off, what are application developers using AI for these days? Well, AI has an increasingly wide range of uses. It began with computer image and processing, such as identifying cats and dogs or postal codes. Now, voice recognition, voice-to-text and text-to-voice are commonplace, and AI is being used to create ever more sophisticated applications. One area that Pingel is excited about is applying AI to help with battery life management, predicting the lifespan of batteries for electric vehicles and consumer electronics.

    “Anywhere there’s data, there’s an opportunity to create models and make predictions based on that data,” said Pingel. This especially rings true for scenarios involving data and visualization. For example, consider visual inspection in a pharmaceutical production line to ensure there are no defects with any pills. Another use case is visual inspections of parts in automotive assembly lines. Applied to these scenarios, AI can remove grunt work and make production safer and more efficient.

    ChatGPT is another example of an AI application quickly becoming mainstream, not only for content creators but for scientific research. For example, ChatGPT can spin up MATLAB code and create functions and unit tests, which can be copied and pasted into IDEs. However, code written by ChatGPT should still be checked and verified by engineers and scientists to ensure its accuracy, said Pingel.

    How Low-Code Lowers the Barrier to Bespoke AI

    Low-code solutions can bring many benefits—these platforms help to speed up tedious tasks and level the playing field for those with different coding abilities and desires, said Pingel. Low-code platforms can also be a great tool for engineers looking to incorporate AI into their business systems. For example, having a low-code method could be helpful in generating code for machine learning algorithms and new models, which can help companies create their own algorithms and incorporate them into their own platforms.

    More people are interested in incorporating AI into their applications, but the use cases are growing increasingly more sophisticated. Businesses may start with simple needs, such as identifying faces or identifying text from handwritten letters. These rely on simple datasets, and algorithms already exist to accomplish most of these tasks. Off-the-shelf AI is helpful at the start, but eventually, companies tend to want to use their own data and settings far more specific to their applications, said Pingel.

    For example, a more niche case could be as specific as identifying cracks in a windshield. Or, in a financial industry, it might be analyzing mass amounts of loan applications to determine approval rates. In these situations, using your own internal data becomes necessary to create AI that understands the situation at hand. Yet, the hard part is assembling this data in a way that is accessible for a model to comprehend. Analysts estimate that upwards of 90% of data held within enterprises is unstructured, meaning it requires a heavy amount of tagging to become usable for AI/ML.

    Low-code is a perfect solution to avoid the point-and-click aspects of cleaning, cropping and structuring this data, said Pingel. This can introduce automation into the process of labeling data while still involving some human intervention to fix mistakes. Machine learning models can take a while to train and simulate, underlining the need for a fast way of iterating on models. A common platform could help companies develop custom machine learning algorithms and models and keep track of experiments. This helps to ensure that the most effective model is used for the final application.

    End Benefits of Democratized AI With Low-Code

    Overall, low-code platforms can provide a number of benefits that lower the barrier to unique AI creations. Low-code platforms like MATLAB and Simulink are useful for structuring and labeling data and quickly generating ML models. Low-code also makes it easier to bridge different programming languages and quickly generate code for a variety of different languages, such as C, C++ and Java.

    Additionally, low-code platforms enable users to collaborate easily with other users. But low-code doesn’t mean no-code, Pingel explained. In fact, “you can learn to be a better programming through low-code tools, too,” she said. For example, it could enhance understanding of the structure and syntax of code.

    In general, democratized AI is an important capability for businesses to take advantage of. Although AI-based solutions are coming to market to displace old tools and processes, there is still an argument to be made for low-code—namely in empowering non-technical users and lowering the barrier to creating custom AI-driven functionality.

  • Fiberplane Adds Wasm Plug-in Framework for DevOps Notebook

    Fiberplane Adds Wasm Plug-in Framework for DevOps Notebook

    Fiberplane today announced it is extending the reach of a real-time collaboration notebook designed for DevOps teams by making available tools for building plug-ins based on the WebAssembly (Wasm) framework.

    Micha Hernandez van Leuffen, Fiberplane CEO, said these plug-ins, dubbed providers, will make it much simpler for anyone to connect the Fiberplane notebook to other DevOps tools and platforms.

    The Fiberplane notebook uses the same operational transformation algorithm used in Google Docs to enable teams to easily collaborate around a document or spreadsheet. The core platform itself is built in Rust and WebAssembly. That approach enables a DevOps team to, for example, write and run PromQL queries and display Prometheus charts alongside other data presented via an integrated graph. Wasm makes it possible to run those providers both in the browser and on a backend platform.

    Fiberplane also announced it is open sourcing its Provider Development Kit (PDK), written in Rust, to enable organizations or third-party developers to create additional plug-ins. A Fiberplane Daemon then executes the plug-ins. The overall goal is to provide an open format based on Wasm for consuming DevOps data, noted Hernandez van Leuffen.

    The full open source Fiberplane WebAssembly-based Plugin System includes a Fp-bindgen to provide the bindings generator for building plug-ins, the Provider Development Kit (PDK), a toolchain for building plug-ins, a set of pre-configured plug-ins for Prometheus, ElasticSearch, Sentry, Grafana Loki, Cloudwatch and a generic HTTP provider, the Fiberplane command line interface and the Fiberplane Daemon.

    Fiberplane is essentially taking the concept of notebooks used for collaboration by data science teams and applying it to DevOps workflows. The goal is to improve DevOps teams’ productivity by reducing the amount of time it takes to sort through multiple types and classes of DevOps data to resolve an issue.

    As IT environments become more distributed in the age of the cloud and microservices, troubleshooting issues now invariably spans the responsibilities of multiple members of an IT team. Fiberplane, in effect, makes it possible to spin up a virtual war room that enables the members of an IT organization to collaborate until the issue of the moment is resolved.

    The speed at which any IT issue is resolved has become a crucial metric. Given the complexity of IT environments, it’s inevitable that incidents will arise. Ideally, IT teams would discover those issues before they disrupt the business. However, in reality, the metrics organizations use to evaluate the effectiveness of a DevOps team are often how quickly IT issues are resolved once they are discovered. The longer it takes for a distributed DevOps team to marshal its resources, the more likely it becomes that team will be perceived as being inefficient—regardless of what level of application availability is actually achieved.

    Most DevOps teams have, of course, implemented a wide range of incident management best practices that create a level of muscle memory for responding to a wide range of issues. The trouble is there will always be unforeseen issues that arise. DevOps teams, as a result, need to find ways to quickly focus on an unpredictable issue that they didn’t anticipate.

  • Graduating From DevOps to MLOps? 5 Tools to Help

    Graduating From DevOps to MLOps? 5 Tools to Help

    DevOps is a set of practices that aims to bridge the gap between software development and operations. It aims to improve collaboration and communication between these two teams and to automate the process of software delivery so that changes can be made and deployed more quickly and easily. This can include continuous integration, continuous delivery and infrastructure-as-code. The goal is to enable faster delivery of new features and improvements to customers while also improving the reliability and stability of the software.

    What is MLOps?

    Machine learning operations (MLOps) is the practice of applying DevOps principles and practices to machine learning projects. It involves the collaboration of data scientists, machine learning engineers and operations teams to streamline the process of building, testing, deploying and maintaining machine learning models in a production environment. 

    The goal of MLOps is to increase the speed and reliability of deploying ML models while also improving the overall quality of the models. This can be done by using tools such as version control, continuous integration and automation of the model deployment process.

    Why DevOps is Not Enough: The Benefits of MLOps

    Machine learning models have unique requirements, such as data privacy, version control and model deployment that are not addressed by traditional DevOps practices. MLOps is specifically designed to address these unique requirements and to ensure that machine learning models are deployed and managed in a production environment in a way that is consistent with other software components. 

    There are several benefits to implementing MLOps in a machine learning project, above and beyond the benefits of DevOps:

    • Improved collaboration: MLOps promotes collaboration between data scientists, machine learning engineers and operations teams, which helps to ensure that models are deployed quickly and efficiently.
    • Faster deployment: By automating the model deployment process and using continuous integration and continuous delivery (CI/CD) practices, MLOps can help to speed up the time it takes to deploy a model to production.
    • Improved model quality: By using version control and automated testing, MLOps can help to ensure that models are thoroughly tested and of high quality before they are deployed to production.
    • Better monitoring and maintenance: MLOps can help to improve the monitoring and maintenance of deployed models by providing better visibility into the performance of models and enabling easy rollbacks if necessary.
    • Better model governance: MLOps can help to improve model governance by providing a clear process for the management of models, including versioning, testing, and deployment.

    5 MLOps Tools to Help Transition From DevOps to MLOps

    Making the move from DevOps to MLOps typically requires dedicated tools that can help manage the complexity of machine learning models and datasets. Here are a few tools, some open source and some commercial, that can help.

    MLflow

    MLflow is an open source platform for managing the machine learning life cycle. It aims to make it easier for data scientists and machine learning engineers to develop, deploy and track machine learning models. MLflow provides several capabilities that can help streamline the machine learning development process, such as:

    • Experiment tracking: MLflow allows data scientists to track and compare different versions of their models, including the code, parameters and metrics associated with each run.
    • Model packaging: MLflow allows data scientists to package their models in a format that can be easily deployed to a variety of environments, such as a local machine or a cloud platform.
    • Model deployment: MLflow provides tools for deploying models to a variety of environments, such as a REST API or a container.
    • Project management: MLflow allows data scientists to organize their work into projects, including multiple models and associated files.
    • Model registry: MLflow provides a centralized model registry that allows data scientists to store and manage models.

    Pachyderm

    Pachyderm is an open source platform that provides data versioning and pipeline management for machine learning and big data workloads. It allows data scientists and engineers to manage, version and collaborate on large datasets and create reproducible and reusable data pipelines.

    Pachyderm uses a version control system (VCS) like git to version data, which allows data scientists to track and collaborate on data sets and enables reproducibility and rollbacks. This allows tracking the changes in the data, seeing who made the changes and when they were made and rollback to previous versions if needed.

    It also provides a powerful pipeline management system, which allows engineers to create, test and deploy data processing pipelines. The pipeline management system is designed to work with containerized workloads, which makes it easy to deploy and manage pipelines.

    Pachyderm is built on top of Kubernetes, which allows for easy scalability and can be run on-premises or on any cloud provider. It also supports data processing frameworks like Apache Spark and Apache Flink, as well as machine learning frameworks like TensorFlow and scikit-learn.

    Amazon SageMaker

    Amazon SageMaker is a fully-managed cloud-based machine learning platform. It provides a suite of tools for building, deploying and managing machine learning models. It is designed to make it easy for data scientists and developers to quickly and efficiently build, train and deploy machine learning models in a production environment.

    Some of the key features of Amazon SageMaker include:

    • Model building: Amazon SageMaker provides a variety of tools for building machine learning models, including pre-built algorithms and frameworks for popular libraries such as TensorFlow, Apache MXNet and scikit-learn.
    • Model training: Amazon SageMaker allows data scientists and developers to train machine learning models on large-scale datasets, using distributed training and automatic scaling to improve performance and reduce training time.
    • Model deployment: Amazon SageMaker makes it easy to deploy machine learning models to production, with a variety of options for hosting models, including Amazon SageMaker hosted endpoints and Amazon Elastic Container Service (ECS) and Kubernetes.
    • Model management: Amazon SageMaker provides tools for monitoring, updating, and maintaining deployed machine learning models, including automatic A/B testing, canary deployments and automatic model rollbacks.
    • Experiment management: SageMaker experiment management is a feature that allows data scientists to keep track of the different versions of their models, the parameters and the metrics associated with each run.
    • AutoML: SageMaker AutoML is a feature that allows data scientists to automatically explore different algorithms and hyperparameters to find the best model for a given dataset.

    H2O MLOps

    H2O.ai’s MLOps is a platform that allows data scientists and engineers to collaborate and deploy machine learning models in production. It provides a set of tools for model development, management, and deployment, as well as monitoring and governance capabilities. 

    It aims to streamline the process of taking machine learning models from development to production while maintaining high standards of model quality and performance. Some of the main features include:

    • Model Development: It provides a set of tools for developing and testing machine learning models, including support for popular frameworks such as TensorFlow, PyTorch and H2O.
    • Model Management: It allows users to organize and manage models in a central repository, making it easy to track versions and collaborate with other team members.
    • Deployment: It provides an easy way to deploy machine learning models in a variety of environments, including on-premises, in the cloud or at the edge.
    • Monitoring: It offers monitoring capabilities that allow users to track the performance and quality of deployed models in real-time and automatically detect and diagnose issues.
    • Governance: It provides governance capabilities to help ensure compliance with regulatory requirements, such as data privacy, and to help manage the risks associated with deploying machine learning models in production.

    Neptune.AI

    Neptune.ai is an open source platform that provides a suite of tools for organizing, tracking and reproducing machine learning experiments. It is designed to help data scientists and machine learning engineers to keep track of their work, collaborate more effectively and improve reproducibility.

    Some of the key features of Neptune.AI include:

    • Experiment tracking: Neptune allows data scientists to track and compare different versions of their models, including the code, parameters and metrics associated with each run.
    • Experiment management: Neptune provides a user-friendly web interface for organizing and managing experiments, which makes it easy to keep track of progress and share results with others.
    • Reproducibility: Neptune provides a way to reproduce experiments by tracking the code, data, environment and dependencies used in each run.
    • Collaboration: Neptune allows data scientists to share and collaborate on experiments with others and provides a way to track who made changes and when they were made.
    • Model management: Neptune provides a centralized model management system that allows data scientists to store and manage their models, including versioning, testing and deployment.

    Conclusion

    In conclusion, MLOps is a practice that applies DevOps principles and practices to machine learning projects to streamline the process of building, testing, deploying and maintaining machine learning models in a production environment. The goal of MLOps is to increase the speed and reliability of deploying ML models while also improving the overall quality of the models. 

    There are several tools available that can help organizations to implement MLOps, including MLflow, Pachyderm, Amazon SageMaker, H2O MLOps and Neptune.AI. Each of these tools provides a different set of features and capabilities, but all aim to make it easier for data scientists and machine learning engineers to develop, deploy and track machine learning models.

    Image Source