Tag: ml

  • How Embedded Analytics Can Transform Future Businesses

    How Embedded Analytics Can Transform Future Businesses

    It’s no secret that big tech companies like Google, Amazon and Netflix are not only leveraging data but also delivering embedded analytics and insights as a standard functionality of their software, services, applications and toolkits. The analytics appear to consumers as recommendations, usage habits and other helpful information. Such insights not only help consumers extract the most value out of the product they are using but also help providers understand how they can provide better services. Businesses that embed analytics to provide a differentiated user experience also have a direct impact on engagement and revenue.

    Have you considered how embedded analytics could differentiate your company’s own software and hardware products? Let’s understand how embedded analytics can help transform technology businesses:

    1. Analytics are expected and welcomed in your products

    By embedding analytics into existing enterprise software and applications, users can drill down into real-time and context-aware data (for example: How busy a location will be at various times of the day) and can benefit from faster data-driven decision-making without worrying about application compatibility. Commonly, software applications provide in-app analytics to delight and serve their customers, while hardware companies can provide helpful stats on a web portal or app.

    2. Embedded analytics is easier than ever today

    Cloud technology has completely disrupted tech-enabled industries. It’s easier than ever for software, apps and hardware to connect to a data repository to save logs that are a rich source of analytics. The invention of the S3 bucket and object store has made it fairly cheap to keep log data. Finally, the transition to platform-as-a-service (PaaS) and software-as-a-service (SaaS) has fundamentally added an “easy button” to tracking consumption patterns and resource utilization.

    3. Embedded analytics helps design better products and services

    Embedded analytics can give businesses more insight into application telemetry (how customers use the data, how they engage with the product) and provide insights into the challenges they face from a usability standpoint. Providers can use advanced analytics to quickly iterate, draw patterns and make predictions, troubleshoot customer challenges, discover new avenues to monetize, detect and mitigate security threats or create entirely new features and applications that put customers on a faster path to improved application experience and better decision-making.

    4. Embedded analytics increases operational efficiencies

    The tech industry is transitioning from traditional monolithic architectures where applications are self-contained to microservices architectures where different components are spun up as needed via API calls. By embedding analytics directly into developer tools, businesses can manage resources without sacrificing availability or performance. Companies can allocate and optimize resources dynamically and understand peaks and valleys in consumption. Businesses must be able to leverage historical and real-time data to build predictive models to proactively identify and mitigate issues with individual components of the application before the customer experiences a problem.

    Embedding Analytics in Your Application

    Before you start embedding analytics in your application, it’s important to choose the right database foundation, as this will support current needs and future goals. Below are the top capabilities that should be standard features in analytics databases:

    1. Handles massive data volumes: The database must scale easily and efficiently with minimal cost using massive parallel processing. Today’s scale might be terabytes, tomorrow it might be petabytes and so on.

    2. Enables speedy data analytics: Speed is a key factor in a positive user experience. Users want instant insights because they don’t like waiting for results. The database should be capable of optimizing load and query performance so that analytics is both timely and relevant.

    3. Embeds machine learning: Machine learning algorithms are critical for predictive analytics. A platform that can embed ML and iteratively learn from new data can make a big difference and add a whole new level of intelligence.

    4. Features native integrations: The more time developers spend figuring out tools and integrations, the less time is spent innovating. The platform must ideally have out-of-the-box integrations with popular applications, data sources (on-premises, cloud, multi-cloud or hybrid), ETL, and visualization tools (e.g., Microstrategy, Looker, Power BI, Qlik, Tableau). It should be flexible enough to support user-defined functions for things that don’t have native integration.

    5. Encourages self-service: The database should make it easy for data scientists to leverage tools and programming languages they already know such as R, Java, C/C++, Python and SQL. Data scientists will also appreciate a platform that has built-in functions for a wide range of analytics (such as geospatial, time-series, event-series, real-time, machine learning, text analytics, pattern matching and regression) and delivers the tools to spot trends, uncover anomalies and anticipate the unexpected.

    6. Supports highly advanced analytics: Many platforms can analyze single tables or do a simple look-up but few have the power to analyze dozens of tables, hundreds of data types, dimensions and attributes from numerous sources over several years of history.

    7. Unified: The database should be able to streamline everything right from data ingestion to exploration to production until delivery. Users should be able to analyze data in its place to always have a single source of truth.

    It’s not hard to visualize that the most disruptive companies of the future will be the ones that harness data as a core asset and embed analytics in their products and services. Embedding analytics requires the right database foundation; to get to the right foundation, IT teams need to cut through the noise and study the fine print. With the proper database, the future is yours to create.

  • Finding Value in AI-Augmented DevOps

    Finding Value in AI-Augmented DevOps

    While artificial intelligence (AI) and machine learning (ML) are emerging technologies, we know they can help an organization parse large data sets and glean actionable insights. But do AI-infused processes actually make a difference in organizations that employ DevOps? The answer is explored in a recent survey by Tricentis. Working in partnership with Techstrong Group, Tricentis surveyed more than 2,600 DevOps practitioners across the globe about their AI-augmented DevOps processes. The results pointed to a bright future in DevOps.

    In this survey, the majority of respondents cited they’re already getting value from AI-powered DevOps processes. A third of survey respondents found AI-augmented DevOps extremely useful today and nearly half found it very useful. Although the technology is still relatively young, a 2021 report by IT infrastructure automation company Puppet Labs found that 83% of organizations are implementing DevOps processes. This rapid adoption speaks to the promise of the technology.

    Interestingly, nearly two-thirds of respondents cited that testing is the area within DevOps where AI will have the most impact. Testing is a major pain point for DevOps organizations because a scaled DevOps environment calls for complex testing scenarios and large amounts of data, and many organizations struggle to scale their automation to the level DevOps requires. Without automation, manual testing can significantly delay releases. In AI-infused DevOps processes, AI can help accelerate the authoring of tests, help testers understand where the risks are, and repair broken tests.

    AI’s Greatest Potential is in Functional Testing

    We typically think of software testing as either functional, where the program enables users to properly execute tasks, or non-functional, where the application is secure, fast and stable. Respondents were asked which disciplines of testing stood to gain most from AI and 65% of them indicated functional testing. Unit testing and UI testing were the sub-categories most often selected.

    The complexity of the functional testing process is where AI shines. Functional testing is open-ended and many permutations are required to ensure proper test coverage. AI steps in and manages those permutations, parsing large amounts of data and providing insights. Additionally, AI can help wrangle production signals that aren’t discovered in a test environment.

    Throughout the survey, respondents specifically noted AI’s promise in regard to UI testing. UI testing is perhaps the most time-intensive and manual type of testing in most organizations. With the many variations of the user experience paths that exist, AI can help mimic the actions of a real end-user, adjusting for application nuances or variations in experience. AI and ML can also boost your build process, increasing the efficiency of repetitive tasks. The ability to quickly identify and resolve issues can improve the effectiveness of your automation.

    While survey respondents see the most potential in testing, they acknowledge that AI-powered DevOps can address other business challenges. More than 40% claim that these processes can reduce the skills gap and empower junior staff to perform complex tasks, improve the customer experience, cut costs, boost innovation and increase efficiency among developers. Nearly 40% also believe that AI-powered DevOps delivers leadership the insights they need for continuous improvement.

    Automation is the Key to Scaling DevOps

    It’s worth noting that among the DevOps organizations that see benefits from AI and ML, most of them are mature organizations; they employ DevOps workflow pipelines, toolchains, automation and cloud technologies, and they’ve automated more than half of their testing. Only 21% of practitioners who are just dipping their toes into DevOps have achieved this level of automation. And even in mature organizations, testing is still the main bottleneck; only 40% of them have automated more than half of their testing.

    The key to scaling DevOps practices is developing test automation that can keep up with an increasing volume and speed of releases. But the road to DevOps isn’t always easy. Practitioners cited obstacles like a lack of tech skills (44%), insufficient budget (25%) and tool selection (19%). Organizations that want to implement DevOps need to rely on external vendors and internal resources to address these challenges and effectively scale.

    Looking to Adapt More DevOps Practices?

    So where does an organization that wants to implement DevOps start? With the big questions. It’s crucial to identify the specific business problem that you’d like to solve. What’s the gain you’re looking for? Where in the software life cycle do you expect to find a benefit? AI and ML can add business value, but they don’t solve every problem across the life cycle.

    After an organization has identified its DevOps goals, its focus should shift to automation. To reap the benefits of AI-powered testing, an organization should automate repetitive functional testing, especially unit and regression testing. Most organizations start by implementing continuous integration and continuous delivery. Once an organization has taken these steps, it will start to amass a large cache of results and artifacts generated from running the test cases, which AI can mine to increase test stability and identify recurring problems. The more you automate, the more you’ll get out of it and the faster you’ll accelerate your delivery.

    Organizations just starting out with DevOps should also understand the distinction between artificial intelligence and ML. Most people refer to ML as AI, but they’re not interchangeable, and most organizations haven’t advanced to the use of AI. A true AI is broadly powerful, while ML technologies are narrow and solve a specific domain problem that a model was trained for.

    When you’re evaluating potential vendors, there are a number of key questions to ask, including:

    • What specific AI technology is the vendor using? Is it an off-the-shelf library or something they built themselves?

    • How are they using AI? Which algorithms are in play and what approach do they take?

    • How do they train their models? Different types of patterns emerge in data sets depending on the industry, the frequency with which data is updated and the application. The data population that the vendor is training on must look like your production data, or the model could take you in the wrong direction.

    • How do they ensure that bias doesn’t come into play? If they train models in a different domain or vertical, for example, you could be an outlier for that vendor.

    • How do they handle failure? How do they monitor for miscategorization and misapplication of the AI in a way that lets you trust the data?

    • Will your data be used to train and improve your learning models? If so, does that raise any privacy concerns? Expectations around data privacy should be explicit and leave no questions.

    • Are there other customers in your vertical with a similar business profile that are getting the benefits you’re looking for? Can the vendor provide examples?

    What Does AI-Infused DevOps Look Like in Practice?

    Consider a financial services organization that has more than 20 applications to manage. With so many applications, it’s necessary to conduct significant UI testing and the tests must be stable. One of the biggest challenges to UI testing lies in locating various web elements; most people start with manual script-based automation, but it can be challenging and time-consuming to create and maintain. Employing AI and smart locators to identify these elements can save time and drastically cut down on UI testing maintenance.

    Survey respondents were asked how they’re currently using AI to augment their testing processes. Thirty-seven percent say they’re accelerating the creation of their automated test cases, while 44% focus their testing on the areas of highest risk. Almost half are able to reduce test case maintenance with self-healing and 43% use AI to identify the root cause of failed tests. Thirty-one percent claim that AI offers insights into test process improvements and 34% use AI to help identify tests to run based on changes within the application.

    Where is AI-Powered DevOps Going Next?

    Going forward, AI will become more and more embedded in testing processes. Great progress with off-the-shelf vendors is being made, and their products are a lot more accessible than they used to be. As the tooling continues to mature, these products will reduce the need for specialized skills. Internal initiatives like data lakes or data warehouses will persist, but there will be less need to build large data science teams to get the advantages of AI.

  • Iterative Adds Registry to GitOps Portfolio for ML Models

    Iterative Adds Registry to GitOps Portfolio for ML Models

    Iterative added a registry for machine learning models to its portfolio of Git-based tools for infusing artificial intelligence into applications.

    Iterative CEO Dmitry Petrov said the Iterative Studio Model Registry is intended to make it easier for application development teams to track which models are being used as the number of applications infused with AI capabilities continues to grow.

    The Iterative Studio Model Registry, accessible via a graphical interface, command line interface or application programming interface (API), makes it simpler to explore models, including history, versions and stages, via a central dashboard. Teams can also identify the experiment that produced the model and track how, when and by whom a model version was created.

    The Data Versioning Control (DVC) platform created by Iterative exposed a Git-like interface to enable organizations to track multiple versions of data sets, models and pipelines across a machine learning operations (MLOps) workflow. That approach enables those models to be stored in a Git repository alongside other software artifacts versus requiring organizations to deploy a separate platform just to store ML models.

    Providers of platforms used by data science teams have been making a case for separate repositories for tracking the artifacts used to construct an AI model. Iterative contends it is more efficient to employ Git repositories where other types of software artifacts are already being stored because, after all, an ML model is just another type of software artifact.

    That approach also makes it simpler for DevOps and data science teams to collaborate because the DevOps team will have more visibility into what AI models will eventually need to be incorporated into an application, noted Petrov.

    The challenge organizations face is the cultural divide between data science and application development teams. Data science teams today typically have defined their own workflow processes using a wide range of graphical tools. However, as it becomes obvious that almost every application is going to be infused with machine learning and deep learning algorithms to some degree, the need to bridge the current divide between DevOps and data science teams will become more acute.

    In fact, DevOps teams should assume that many more AI models are not only on the way but that those models will need to be continuously updated. Each AI model is constructed based on a set of assumptions; however, as more data becomes available, AI models are subject to drift that results in less accuracy over time. Organizations may even determine that an entire AI model needs to be replaced because the business conditions on which assumptions were made are no longer valid. One way or another, the updating and tuning of AI models are likely to soon become just another continuous process being managed via a DevOps workflow.

    No IT organization can, of course, manage what it can’t track. Registries have become an essential component of a DevOps workflow as the volume of types of artifacts that make up application environments has increased. It remains to be seen to what degree ML models become just another software artifact but one way or another someone in the IT organization needs to know what model is being employed in which application for what purpose.

  • VW CEO Fired for Dev Fails | Fiber Shortage Hits | Google Fires Blake Lemoine

    VW CEO Fired for Dev Fails | Fiber Shortage Hits | Google Fires Blake Lemoine

    Welcome to The Long View—where we peruse the news of the week and strip it to the essentials. Let’s work out what really matters.

    This week: Herbert Diess is out at VW because software is hard (yo), fiber optic cable is hard to find, and the guy who said LaMDA was sentient is dumped by Google. (more…)

  • Google FLoC is Dead | Meta AI Supercomputer Lives | ARM Deal is Dead

    Google FLoC is Dead | Meta AI Supercomputer Lives | ARM Deal is Dead

    Welcome to The Long View—where we peruse the news of the week and strip it to the essentials. Let’s work out what really matters.

    This week: Google’s FLoC proposal is dead, Meta/Facebook is buying RSC—a huge AI supercomputer, and Arm “will IPO” instead of selling to Nvidia.

    1. Forget ‘Federated Learning of Cohorts’—Long Live ‘Topics’

    First up this week: Google bows to the inevitable. Third-party cookies will soon go away, because people are fed up with being tracked. But Google’s FLoC proposal wasn’t the answer.

    Analysis: FLoC is a flop

    Google doesn’t want to invade your privacy. But Google does want to sell well-targeted ads. The key difference between FLoC and Topics is that FLoC buckets were automatically generated—and hence opaque, risking overlap with “protected categories”—e.g., race or gender. But the topic classifications in the Topics API are curated and have user-visible, descriptive names.

    Frederic Lardinois: Google kills off FLoC, replaces it with Topics

    Google’s controversial project for replacing cookies for interest-based advertising … is dead. In its place, Google today announced a new proposal. … Your browser will learn about your interests as you move around the web.
    …
    When you hit upon a site that supports the Topics API for ad purposes, the browser will share three topics you are interested in. … The site can then share this with its advertising partners to decide which ads to show you. … Users will be able to review and remove topics from their lists [or] turn off the entire Topics API.
    …
    It’ll remain to be seen if other browser vendors will be interested in adding the Topics API. [But] they all quickly turned a cold shoulder to FLoC. … The plan is to start trialing the Topics API at the end of this quarter.


    A minor tweak and rename? “Yes and no,” equivocates mananaysiempre:

    Yes, it still amounts to building user-tracking functionality into user agents. No, it’s not the same approach.
    …
    FLoC was doing unsupervised clustering of users, whereas Topics allocates users into predetermined clusters. This makes it potentially more transparent and also seems designed to work around the “protected categories” objection.


    But Google has a long way to go to convince garett_spencley:

    I’m glad FLoC is being ditched, but as an end-user I’m still not asking for this Topics nonsense. It doesn’t solve any problem that I have. It solves problems that the advertising industry has.
    …
    I’ll choose to use browsers that don’t support this.


    2. Machine Learning at Metaverse Scale is Expensive

    Zuckerberg’s new toy is RSC: A $400 million Nvidia-based AI supercomputer. And that’s just Phase One.

    Analysis: 3.8×1018 integer operations per second is … a lot

    If you rely on Open Compute Project systems for your DevOps, know that it’s no longer the bee’s knees over at Hacker Way. Meta/Facebook is investing eye-watering sums into a custom configuration of off-the-shelf hardware.

    Timothy Prickett Morgan: Meta Buys, Rather Than Builds And Opens, Its Massive AI Supercomputer

    If you thought it took a lot of compute and storage to build Facebook’s social network, you ain’t seen nothing yet. … To build the Metaverse … is going to take an absolutely enormous amount of supercomputing power.
    …
    But what is a surprise, given Facebook’s more than decade long focus on … opening up its hardware designs through the Open Compute Project, is that the … Research Super Computer [RSC] system is built from … commercially available … servers, storage, and switches. [It] probably comes down to … working a deal with some vendors who want the Facebook publicity and are willing to work the price and the parts availability.
    …
    [So Meta bought] 760 DGX A100 systems from Nvidia—which each … have a pair of AMD “Rome” 64-core Epyc 7742 processors … 1 TB of main memory … eight A100 GPU accelerators [and] eight 200 Gb/sec HDR InfiniBand controllers—all-flash storage from Pure Storage, and … caching servers from Penguin Computing. … At list price this machine would cost around $400 million … Phase one of the RSC machine is rated at 59 petaFLOPS … (FP64), and 118.6 petaFLOPS FP64 using the Tensor Core matrix engines. … With INT8 processing on the Tensor Cores, which is needed for AI inference, the phase one machine is rated at 3.79 exaOPS.


    How much??? It doesn’t seem like news to kleiba:

    I used to work at a university where my professor had been in automatic speech recognition for a long time, but basically gave up on that line of research about 10 years ago because he figured that universities simply cannot compete budget wise with the big industry players. I suppose the same will soon be true for most ML-related areas of research sooner or later.
    …
    Already, a substantial amount of research innovation in [Natural Language Processing] and [Computer Vision] has been coming from big companies in recent years. Of course there is a discussion to be had about what that means for society at large.


    But 3.79 exoOPS? That’s the way the wind is blowing, according to sungazer:

    Give it 15 or so years—this level of computation will be on your desktop. If desktops still exist.
    …
    Or desks.


    3. Arm Bid: No Legs

    Speaking of Nvidia, as I’ve said before, the Arm merger really shouldn’t go through. And it’s looking less and less likely. ARM chips are an increasing fixture in the data center—especially those that value “performance per Watt.”

    Analysis: ARM M&A DOA—IPO PDQ

    A veritable hornets’ nest of Bloomberg journalists have put their names to an anonymously-sourced report. The story argues that the proposed Nvidia buyout of Arm from SoftBank Group is dead. Next stop: IPO.

    Ian King, Giles Turner, Peter Elstrom, Dina Bass, David McLaughlin, Ruth David and Dinesh Nair: Nvidia Quietly Prepares to Abandon … Arm Bid

    According to people familiar with the matter … Nvidia … doesn’t expect the transaction to close. … SoftBank, meanwhile, is stepping up preparations for an Arm initial public offering.
    …
    If Nvidia manages to get the deal over the line, it would be a massive coup. … But it will be an uphill fight. … The world’s biggest tech companies rely on Arm technology, and they fear they could lose unfettered access under Nvidia.
    …
    Within SoftBank, there are factions. [And] the ordeal has created divisions within Nvidia. … “We remain hopeful that the transaction will be approved,” a SoftBank spokesperson said in an emailed statement.


    Post-Brexit Britain would no doubt like to keep Arm independent. Here’s simonh:

    Arm has I think ⅔ of its employees in the UK—mainly Cambridge—and they’re expanding the campus there. The Semiconductor Physics Group at the Cavendish Laboratory at the University is at the cutting edge globally with 18 PhD projects currently.
    …
    Guess where scads of Arm’s top scientists are recruited from? Arm and the University are joined at the hip. That company’s not going anywhere.


    More broadly, it’s good news, thinks hatchet:

    Hopefully this means NVIDIA will focus on RISC-V instead. This will be good for everyone.


    The Moral of the Story: Zeal should not outrun discretion.


    You have been reading The Long View by Richi Jennings. You can contact him at @RiCHi or tlv@richi.uk.

    Image: Reed Geiger (via Unsplash)

  • Driving Software Delivery Automation With AI/ML

    Driving Software Delivery Automation With AI/ML

    What if we could use an artificial intelligence (AI) engine to predict the next lines of software code to build? What if machine learning (ML) could trigger the creation of automated test cases to validate our solutions?

    Automation has been a focus for the information and communications technology (ICT) industry for years, and there are some lessons that could apply to the software developer community, too, as well as other verticals. Today, more streamlined execution of operational and software delivery processes, such as continuous integration, continuous delivery and continuous deployment (CI/CD) in app development, continue to drive new levels of cost reductions, allowing products to get to market and update faster. 

    However, with the increased amount of data comes more complex ecosystems and the need for the next level of automation. Hyperautomation, or the use of AI/ML to further streamline processes, could build on these CI/CD processes, revolutionize the way multiple industries deliver software and optimize operational processes.

    Today’s Network Complexities and Challenges

    In the telecommunications industry, disruptions from hyperscalers and other innovative players, as well as increasing levels of investment in 5G technology, are forcing the hand of communications service providers (CSPs). This is causing them to turn to automation to improve their levels of operational efficiency and innovation to keep up with the need to push frequent updates. 

    However, challenges remain in the form of:

    • Total cost of ownership and operating costs: With more frequent needs for upgrading existing networks, additional levels of automation including AI/ML and hyperautomation are required to manage and maintain today’s complex networks. 
    • Time to market: In an increasingly competitive market, the CSP’s ability to react to market needs and/or new marketing ideas and to rapidly launch those needs and ideas to their end users and enterprise customers is a key competitive differentiator. 
    • Customer experience: Digitalizing business processes, providing an increased level of self-service and seeking ways to optimize customers’ overall journey remains an important need.
    • Employee productivity: In an effort to increase revenue, more efficiently drive their networks and improve end user customer experience, CSPs also need to increase employee productivity. They can do this by engaging their employees to solve bigger problems. As a result, the simplest and most repetitive low-level tasks can be automated.

    The Power of AI and ML

    The automation journey started with breaking down complex processes into smaller parts and automating discrete parts using scripts, runbook automation (RBA) and/or robotic process automation (RPA). 

    With the introduction of AI emulating human intelligence through mechanical and computational processes and ML using algorithms that learn from data to make predictions and decisions, CSPs can now bring automation and hyperautomation to the next level. 

    The potential use cases for the delivery and maintenance of telco software solutions are unlimited, and many of these lessons can be applied to other sectors. Some examples include:

    Autonomous Fault and Anomaly Detection for Cloud-Native Solutions

    Today’s network solutions are deployed on containers and microservices. The identification of issues and assurance of service quality becomes increasingly challenging with thousands of workloads running in parallel. Container failure can cause severe service disruptions. In fact, a single service failure or abnormal behavior could have a butterfly effect on the overall system and negatively affect customer satisfaction. 

    A proactive, AI-based monitoring solution can continuously search for anomalous behavior in the environment. Different container metrics and logs can automatically analyze and detect problems proactively and help find the right corrective actions for the application workflows. This is based on identifying anomalies in the container orchestration metrics related to pods, workloads and namespace. These techniques enable a more reliable and predictable application runtime environment with higher service quality and faster identification and resolution of system faults and, ultimately, a better overall customer experience

    Machine Learning-Triggered Closed-Loop Automation

    Large clusters of compute, storage and networking resources are managed throughout a data center to enable the deployment of custom workloads. Due to the inherent nature of applications running on the virtualized and cloud infrastructure, there are usage highs and lows which need constant infrastructure component deployment or reconfiguration of the cluster to ensure cost-efficient service availability. 

    A machine learning-based solution can detect anomalies in the infrastructure. Specifically, with closed-loop automation, enterprises can bring in a layer of intelligence over the current setup to monitor application performance and infrastructure metrics and identify anomalies that require immediate system attention. If those are found, they can trigger the automated addition of capacity to the infrastructure. When anomalies have subsided and once the utilization is capable of being handled by the original capacity, the solution can consistently monitor metrics to roll back the virtual or cloud-native functions. This allows for an efficient and fully automated use of network capacity, minimizing human oversight.

    Smart Code Recommendations

    One of the most clear-cut ways tools can reduce the effort of software development for programmers is through code completion. AI-powered code completion is a convenient way to access descriptions of functions and their parameter lists. These solutions also speed up software development by reducing keyboard input and the necessity of name memorization. They also allow users to refer less frequently to external documentation, as interactive documentation appears dynamically in the form of tooltips. AI-powered code completion can also use an automatically generated in-memory database of classes, variable names and other constructs or references based on machine learning to open code repositories.

    AI-Driven Test Recommendation

    Significant time is spent understanding solution requirements and identifying critical scenarios for testing. This is usually followed by a manual test plan creation and execution. To solve this, AI/ML techniques can be applied to identify what tests will need to be done during each cycle of the software development stage, including automating the analysis of required documentation. This can be used to increase overall test-cycle time and improve productivity. 

    AI-Enabled Test Automation

    API testing is critical for validating application logic and needs to go beyond the GUI layer to validate the core of an application. This implies complexity because the protocols are usually difficult to understand, the API inventory is usually large and sequencing of the API calls is critical to a successful test specification and execution. With AI/ML and natural language processing techniques, the analysis of the solution test plan can be automated to obtain recommendations on what test frameworks to utilize, identify and suggest the parameters for each test case, identify the API call sequence and automate the development of test cases based on reusable functions. This can increase overall quality of service, simplify change management, improve the productivity of the testing team and offer more reliable test cases.

    Test Failure Analytics

    During execution of automated test cases, multiple failures can be observed in various logs. Manually analyzing the logs and correlating them with the application issues can be time-consuming. Also, the identification of unreliable tests impacting system stability is complex. 

    AI/ML techniques such as vectorization, K-means clustering or topic modeling can be applied to automate the collection of test logs across applications and label failures including the type of cluster failed tests. The test subject matter expert (SME) is provided with a failure analysis report summarizing issues and problems automatically from many logs across applications, simplifying the identification of issues in test automation areas or scripts. These techniques can significantly increase test productivity and overall troubleshooting time.

    Accelerated Root Cause Identification and Analysis (RCI&A)

    When a problem occurs in a telco network, it tends to cause a ripple effect, affecting other integrated services and creating an alert storm. In such scenarios, root cause identification can become complex and cumbersome. Event aggregation and synchronization are needed to reduce environment complexity and variance. 

    Machine learning weighted analysis techniques can be used to determine the influence individual IT metrics have in affecting the overall health and performance of hosted applications and services so that pinpointing the root cause of any service impact in real-time can become easier for the team operating the network. Efficient AI/ML techniques for accelerated RCI&A can lead to significant timeline reductions. 

    AI/ML Driving Operational Innovation and Market Agility

    In telco, there is great potential for AI/ML-driven use cases to further improve the cloud-native journey and enable an IT-like software life cycle management process for the solutions of the future.

    More broadly, highly automated and self-healing networks will drive higher levels of efficiency and customer satisfaction. This includes faster testing with less resources; more efficient utilization of network resources and cloud infrastructure; faster time-to-market with more software releases per year with similar capacity as well as an increased quality for faster and more agile monetization of the CSPs capabilities.

  • DevSecOps Implementation: Intrusion Detection

    DevSecOps Implementation: Intrusion Detection

    Originally, this series was just going to be four articles on the DevSec side of DevSecOps. There are many reasons for this, but primarily because that side is cleaner. The other reason is that these topics are beyond the work we were doing at Accelerated Strategies Group.

    But we’ve had a number of requests to continue, and I’ve used many tools in the space, so on we go.

    Intrusion Detection

    Intrusion detection (IDS) started because attackers were getting through firewalls, and, once inside, seemed to have free rein. Intrusion detection wanted to offer another option for detecting and stopping attackers by watching things inside the firewall for indications your organization had been compromised. Relatively quickly, intrusion detection was supplemented by intrusion prevention (IPS). The idea of prevention was to proactively make changes based upon known issues to keep intruders out. Intrusion detection offered information about what paths attackers were exploiting, and intrusion prevention would plug them. Note that IDS and IPS are often further subdivided into host-based (HIDS) or network-based (NIDS) systems. Since their role – detection of intruders – is the same, for this overview we will discuss them together. A serious deep dive into IDS/IPS should look at HIDS and NIDS separately.

    In the DevSecOps space, some have folded these into security monitoring, but they are still performing the same basic tasks – just much better. In fact, if you found intrusion detection to be limited and clunky to work with because of rules maintenance and fingerprinting issues, it’s probably time to take another look. Here’s why:

    Problems With Traditional Intrusion Detection

    The problems with pure rules-based intrusion detection were two-fold.

    First, rules were hard and fast.

    “Look for X and Y and Z. If all are present, raise an incident.” This tended to result in only two types of rules – ones that were too narrow, and thus brittle, or ones that were too broad and that buried IT/security staff in alerts.

    Second, those rules were outdated before they hit the network.

    The old adage “You are always fighting the last war,” (meaning you are doing what worked the last time you won) is more true in IT security than almost anywhere else. For a rule or a fingerprint to be assembled, it has to have been seen. That means we were perpetually protecting against previously successful attacks and doing little against emerging threats. Security vendors took a lot of steps before coming to our current solution, which is a great step in the right direction.

    Attacks against IT infrastructure share a lot of commonalities. Purely rule-based systems counted on tribal knowledge to code those commonalities into rules. It was clunky and imperfect. Templates helped, by adding common items based upon the type of rule you were creating, but, like rules, they tended to be static and old news.

    Enter Machine Learning

    Bright people with an eye on the ever-changing environment knew that we needed something that could adapt to multiple attack forms/vectors and draw the type of conclusions that security analysts were tasked with. The ability to reduce the number of alerts that get raised is pretty close to miraculous. Anyone with a web presence sees thousands of security events a day, as attack bots and scanners automate chunks of the attack matrix. Some kind of solution to weed out the false information is nearly mandatory, as no organization has the manpower to monitor at those levels and reasonably follow up on threats.

    Machine learning (ML) makes connections between alerts – that might not be a big deal in isolation – and other events that, together, make a case for human intervention. Added to fingerprinting and rules, it is a powerful addition. While a successful attack would require going through rules and making certain you covered the behavior/changes found in the new attack, ML can take that information in and adapt automatically.

    This is cranking through volumes of data we couldn’t even imagine a decade ago, and finding anomalous behavior, correlating it and alerting where necessary. How much it reports up the chain is generally configurable, allowing an organization to tighten the flow of alerts that analysts have to deal with.

    Go With the Flow

    More recently, flow evaluation has become a positive step in the monitoring process. If this connection/user moves from A to B to C, it might be an attack. This allows for another way to rate suspicious activity and raise it to an analyst’s awareness.

    An increasing number of vendors are moving the analysis portion of these activities into their hosted environment. This is a “great strength/great weakness” solution – putting the data points gathered from your system into their cloud allows them to correlate events and improve all customers’ responses. The other side of the coin is that you are paying them operational costs each month, and for most vendors, you’re charged based on how much of the service you use. This has budget implications. It also assumes you have connectivity to their service. If you don’t, for any reason, no analysis will be done on collected data. A careful rollout and contingency planning are in order for this scenario to be successful.

    Integration with security information and event management (SIEM) systems is ubiquitous. We’ll talk about SIEM next; I just wanted to include here that IDS/IPS vendors are feeding SIEM.

    IDS isn’t perfect, and there are a ton of other things out there to help, but it’s a step in the process, one you should consider as you gear up your DevSecOps to professional levels. Some of you are using the tools, but don’t have them integrated into your DevOps systems. This is a nudge to do so. If resolution of an attack automatically creates tickets for Dev to make the code (or ops to make the config) more secure, it absolutely is a plus to the organization responding proactively.

    Keep kicking it. These are just tools, and they’re useless without you. Figure out what your organization needs, and update those needs regularly as the environment (both attackers and architecture) continually changes around you.

  • A Buyer’s Guide to AI and Machine Learning

    A Buyer’s Guide to AI and Machine Learning

    B2B software sales and marketing teams love hearing the term “artificial intelligence” (AI). AI has a smoke and mirrors effect. It sounds impressive. But, when we say “AI is doing this,” our buyers often know so little about AI that they don’t ask the hard questions.

    In industries like the DevTools space, it is crucial that buyers understand both what products do and what their limitations are to ensure that these products meet their needs. If the purpose of AI is to make good decisions for humans, to accept that “AI is doing this” is to accept that we don’t really know how the product works or if it is making good decisions for us.

    When we’re in the buyer role, we often don’t hold ourselves responsible for understanding AI and machine learning (ML) products because these technologies are intimidating. They’re incredibly complex.

    This article addresses the limitations of AI and ML, so software buyers can ask the right questions to understand what they are buying. 

    The Test Oracle Problem

    One limitation of some AI or ML products is that for certain applications of the technology, there is no source of absolute truth to compare against the accuracy of the output. For example, neither humans nor machines know how to produce the perfect set of end-to-end tests for any given application. This is the test oracle problem: there is no objective standard of truth. No one wants to introduce this kind of uncertainty into their sales process. Yet, our buyers deserve well-informed answers about our products.

    As a buyer, you need to understand the intended advantage of your seller’s AI product before making a purchase decision. Is it meant to make a decision that is more accurate—against an objective standard—than a human? Is it meant to make a faster decision with less cost? Or introduce an alternative methodology that uses new data in a new way? Answers to these questions influence how you will use the product and what value it provides.

    AI Versus ML

    Though AI is commonly accepted as “any machine that uses math to make decisions,” true AI is self-taught. AI has a neural net that mimics neurons in a human brain which allows it to teach, update and evolve itself. Because of this, true AI is difficult to build and is often experimental rather than commercial.

    More often, what’s being described when we say AI is actually ML. ML is human-taught: Machines learn through human feedback using a probabilistic decision-making process that improves via ongoing correction. Machines take in data, run algorithms against it and output a decision — or series of assertions — based on probabilities. Humans correct the machine by telling it whether it was accurate in its assessment, and the machine updates. As it receives accuracy feedback, machines learn to make better decisions. And because ML is based on probabilities, it will sometimes make the wrong decisions.

    Based on how you plan to use a product, you need to determine how rigorous its accuracy needs to be. How often a machine can make the wrong decisions and still serve its purpose is application-specific. Self-driving vehicles must be nearly perfectly accurate to be adopted. Paralegal ML toolsets likely need to be less accurate. How accurate does your product need to be?

    Asking the Right Questions

    Regardless of how you plan to use a product, it’s important to ask the right questions to understand the product and build resiliency around its accuracy levels. The next time a seller tells you “AI is doing this,” you can ask the following: 

    1. Is this product an ML product? Does it need to be ML to get a meaningful result? To be ML, a product needs to learn through human feedback, not just make decisions using probabilities. Do you just need a product that uses logic to make decisions, or a product that improves in accuracy over time?
    2. How is the accuracy of this product calculated? You won’t know if the machine is more accurate than humans if you don’t know the conditions used to calculate accuracy. If a machine is 30% more accurate than humans, who assessed this accuracy and how did they determine this?
    3. How do you know when the product makes the wrong decisions? Any ML product will sometimes produce the wrong output. Typically, a seller’s most successful customers have already adopted business processes to build resiliency to this wrong output. If so, the seller can help you adopt them as well.
    4. In its current state, how often does the product make the wrong decisions? Knowing the frequency of mistakes and the stakes of those mistakes will be crucial to deciding how you use the product, and whether it’s safe to do so at this stage in its development.
    5. How many teaching hours have been put into this product? This number will provide a simple approximation of how much effort has gone into making the product more accurate. A low number can be fine, depending on the application.
    6. How does my usage improve the accuracy of this product? As a buyer, you are an integral part of the machine testing and teaching process. You should be willing to use your data to improve their accuracy, because you want these products to improve in the future.

    Why Knowing AI and ML Matters

    Not only is there a lot of “AI” that isn’t AI, but there is also algorithmic technology that isn’t ML. It is thus essential for buyers to know enough to ask the right questions and understand how these products make decisions.

    There are limitations to all ML products, though the limitations differ by product and the way the product is applied. When a product’s accuracy levels are unknown, all you can do is ask if its methodology is valid for decision-making: Does it have access to better data than humans? Can it make smarter and faster decisions than humans with this data? If the answer is yes, you should consider buying the product rather than having your people do the work.

  • DevOps Chat: Increasing Delivery Velocity, with Launchable

    DevOps Chat: Increasing Delivery Velocity, with Launchable

    Jenkins founder Kohsuke Kawaguchi (KK) and respected DevOps veteran Harpreet Singh have launched a new company called Launchable.

    Launchable aims to help you improve your delivery velocity by prioritizing and applying some ML and intelligence to testing. There is still much work to be done but with these two industry luminaries behind it, the expectations are high for something impactful.

    The VC community and a blue chip list of investors are already on board.

    Yes, this means KK is no longer at CloudBees (several CloudBees execs are investors in Launchable) and Jenkins, while he is still an advisory. But, with Jenkins now in the capable hands of the CDF and Linux Foundation, and CloudBees well established, he felt this was a good time to pursue an issue he thinks needs to be solved.

    Hear from KK and Harpreet themselves in this great podcast.

    Transcript

    Alan Shimel: Hey, everyone, this is Alan Shimel for devopsy.kinsta.cloud, and I am here, you’re listening to another DevOps Chat. We have some really great breaking news on this DevOps Chat. I’m really happy to have, snagged a dynamic duo of industry veterans in the DevOps space who have just started what promises to be perhaps a game changer in the DevOps industry. But they’re gonna tell us more about that, I don’t know.

    Let me introduce you to Harpreet Singh.

    Harpreet Singh: Hello.

    Shimel: Hi, Harpreet, how are you?

    Singh: Very well, thank you.

    Shimel: Thank you. And Harpreet is co-CEO of the new company, and his co-CEO is none other than my friend, Kohsuke, K.K., founder of Jenkins, CloudBees and all of that good stuff. K.K., welcome.

    Kohsuke Kawaguchi: Hello. Thanks.

    Shimel: Okay, so, let’s talk about the big news. We have a new company in the DevOps space called Launchable.

    Kawaguchi: Yeah.

    Shimel: Is that—and the both of you are co-CEOs?

    Singh: That’s right.

    Shimel: And, of course, with your track record, I’m sure, you know, it wasn’t as hard as it is for a first time entrepreneur, but Launchable’s already raised some venture capital to help it get off the ground, including some high profile angels.

    Kawaguchi: Yeah. We’ve been very lucky.

    Shimel: Talk a little bit about it—who’s involved, who’s put some money in?

    Kawaguchi: So, there are two big institutional investors. One is Barclays Ventures, and that comes from Harpreet’s long relationship with their partner, Gomesh. I think, in fact, your wives know each other. Like, it almost feels like an insider. [Laughter]

    And then another VC is the unusual ventures, and that’s started by Unusual Ventures, and that’s started by John Vrionis. And John actually has been also an investor at CloudBees, so his people have seen us in action and know us well enough so, you know, you get their validation and then you endeavor it. It’s been—I’ve been truly grateful.

    Shimel: Sure. Well, I’m familiar with Battery back in my day, it was a guy named Gerry Conlon out of Battery Ventures. I don’t know if either of you know Gerry, but he was instrumental early on in Battery. But yeah, they’re a great company. It sounds like two, you know, kind of blue blood venture capital firms in there.

    There’s also some high profile angels including Sacha—Sacha Labourey, CEO of CloudBees, right?

    Singh: Yeah, yeah. So, I think what was exciting for us as we were talking with people was just, you know, the phenomenal backing that we got from investors as well as—you know, these marquee investors as well as individual angels, right? So, we both have been at CloudBees.

    So, I don’t like to think of ourselves as first time entrepreneurs, because we built CloudBees from less than 10 people to whatever it is today.

    Shimel: Yep.

    Singh: I had a sojourn to Atlassian, where I spent about a year and a half really learning from that phenomenal company. So, we were very lucky when people from CloudBees like Sacha, Michel—Michel Goossens, who was the VP of sales there at some point came in and invested. Bob Bickel, who’s been an industry veteran invested in us. And on the Atlassian side where the CTO is Sri Viswanath and Noah, who’s the head of tech teams, which is Jira, Bitbucket, Opshimi and all these comfortable investing in our capabilities and our mission.

    Shimel: Absolutely. And make no mistake, you’re not first time entrepreneurs, because first time entrepreneurs don’t raise money that easily, guys.

    Singh: [Laughter]

    Kawaguchi: [Laughter]

    Shimel: I mean, I think there’ll be a lot of people who listen to this podcast who are first time entrepreneurs and say, “Man, if we only had their background and their resume and their experience, it wouldn’t be as hard as it is.” I think a lot of first time entrepreneurs struggle—struggle with doing that.

    But that’s great. It’s always good to have money in the bank when you start a venture, because it’s needed, but as we were talking about off camera a little bit, I’ve never met a successful startup entrepreneur who did not think what they were doing was in some way changing the world, making it better.

    And so, I wanted to ask each of you individually—what really got your juices flowing? What really got you excited about what you guys are doing at Launchable?

    Kawaguchi: Sure. Alright, so I’ll start. So, you know, one of the things I started—I created this sort of thinking and I felt like I put the state of the automation and the software development power to the next level in the world. And this automation, they produce a lot of data. And what I realized after a while was that, you know, in the rest of the world, in the rest of the business world, data is increasingly being used effectively to drive decisions and make people more efficient and so on.

    But whereas in the software development, I feel like that actually hasn’t really happened. And I felt like this was a great shame, one as somebody who, you know, helps people produce a lot of data with these automation systems, but also, you know, we are the number people, we are the data people, right? We are the pride of ____, at least I think of it that way.

    Shimel: Mm-hmm.

    Kawaguchi: Yet, everywhere I look, I feel like we are making decisions, engineering decisions based on gut and instincts, and this can’t continue. And so, that’s what I kind of started thinking about after a while, and then this idea kind of grew and grew on me. So, yeah—so, that’s probably where my, how my juices got flowing.

    Shimel: Excellent.

    Singh: Yeah, very similar. I think Kohsuke and me used to sit next to each other in the same office room at CloudBees and we used to talk about this all the time. We got the opportunity to see sales and marketing really be completely numbers driven, and we would come back and say, “Oh, if we could only do this on our engineering product side of the house, it would be so much better.” And the answer always was, “Well, it’s a craft.” And we think bringing data in there is important.

    Shimel: Right.

    Singh: And so, that’s sort of what got us both going. My personal anecdote was, after I left and moved to Atlassian, I was talking to somebody out in my customer base there who had this challenge where their delivery velocity was slow and, you know, their aim was to put a few engineers to go in and look at it and how to improve it. And having the DevOps background and you understand CI/CD, I felt like—there has to be a better way to do this than just, “let’s allocate a few engineers and see what comes out of it.”

    Shimel: Right.

    Singh: So, that’s sort of the problem space, and that’s what excited us.

    Shimel: Excellent. And it is, it’s almost ironic or oxymoronic, even, when  you think about it, right?

    Singh: Right.

    Shimel: Because—so, my background’s in information security, right? And one of the things we have seen in security is information overload, like data overload.

    Singh: Right.

    Shimel: The noise to signal ratio. But there have been—there have been a lot of security startups over the last, let’s say five years, seven years, who have really tried to fine tune that dial to get more signal to noise, to be able to automate going through the mounds and mounds of log data, for instance, right?

    Singh: Mm-hmm.

    Shimel: So, you get a company like a Splunk that can just suck a lot of data in from everywhere. And then try to figure out, in an automated way—because there’s just too much to go through by hand—

    Kawaguchi: Exactly.

    Shimel: What’s important from a security point of view or maybe even from a network operations point of view. But as you guys said it, I’m thinking—we don’t really do that with our software development, do we?

    Kawaguchi: Yeah. And I mean, when you describe it like, it might sound a little abstract program, but idealized, it’s actually connected in so many ways to frustrations that I have every day as a developer, right?

    Let me give you an example. You know, I was working on this Jenkins project, and it’s a sizable project with lots of test cases, which is great, but what that means is that every time when anybody wants to make, like, a ____ change, the first thing that happens is, the CI system comes in and they run the whole one hour test cycle before the code review happens, and then somebody makes some observations or a suggestion that, “Oh, you should change this part to something else.” And then you follow that and then another whole hour of testing. And then that’s basically, like—two hours is a long time to wait if you get some change in, actually. And that happens all day in that area.

    Shimel: No, no—I mean, so, let me be a half glass full kind of guy. It’s better than—the one hour is better than the one day it used to be before we automated some of this stuff, right?

    Kawaguchi: That’s true.

    Shimel: I remember back in my StillSecure days and we would—we didn’t even have a big enough infrastructure to do the testing, we would have to outsource, like, unit testing and all this stuff.

    Kawaguchi: Yeah, so, let’s talk about that one big thing, right? Because that’s actually probably an even bigger program. And you said that that’s before automation, but I’ve seen teams every single day that actually have tests that run for more than a day, you know, [Cross talk].

    Shimel: With automation?

    Kawaguchi: Yeah, like an auto maker company putting a car together, a lot of big software. If you test them all, it’s in that cycle for weeks. So, there are a lot of pieces of software getting assembled together into something big, and that test gets even bigger.

    So, what I started doing is, like—okay, so, every time you do this, it’s not like everything is changing all at the same time, right? There’s only small parts of the system it’s changing, and then when we re-run the whole test cycle. So, what if we could ____ the right subset for that test to run? They’re most likely detached programs, so they’ll not only finish quickly, but they also run cheap, and then you still get, like, a pretty sizable, meaningful feedback. You know, you’re not gonna get 100% coverage, but you might get 95%, and that’s plenty good, and that cuts the execution down in, let’s say, half.

    And so, these are the kind of things I thought the data would actually enable, and I’ve seen some prior work, I’ve talked with teams who seem to be on the cusp of doing these things, and I felt like—yeah, this clearly makes sense. More and more people need it, and I need it. And so, that sort of became the founding idea behind Launchable.

    Shimel: So, that was the itch you were looking to scratch here, as I say, right?

    Kawaguchi: Yeah, right.

    Singh: And you asked, like [Cross talk]—

    Shimel: I’m sorry, go ahead.

    Singh: Yeah, and as we were sort of thinking about this, right, we identified this problem and we’re thinking about, like—so, what’s the company’s mission, right? And we realized that every sort of software teams in the future or even today will drive every significant human achievement there. It’s really hard to think about anything that’s not being software delivered.

    And our mission now is to take this artisan teams and actually make them data driven scientists to drive their velocity, you know, and deliver software confidently. So, that’s sort of our big itch and our big mission.

    Shimel: Absolutely, got it. So, let me play devil’s advocate a little bit. You know, one of the things we always used to hear—I heard it in security, I heard it in QA—was, “Look, we don’t have enough time to do a full test, full testing. Let’s just test 80%, 70%, 90%”—take your pick, I don’t care what number you pick.

    You know, Murphy’s Law says whatever you don’t test, that’ll be the thing, somehow, that breaks, right, or that causes a problem. And when we say, “Oh, my God, because we didn’t test that”—why didn’t we test that?

    Kawaguchi: Right. So—

    Shimel: “Why shouldn’t it get done?” Right.

    Kawaguchi: Yeah. So, if you think of testing in isolation, then yes, that’s naturally a concern, because, you know, we need every help and every confidence we can get out of the test. So, like, why would you skip that? That doesn’t make sense.

    But I think where it does make sense is actually, if you think about it, the software development life cycle is composed of a lot of different tests at a lot of different points. So, collectively, they should be seen as a whole system, right? I mean, this is things like—so, the code review, right? If that’s the only point in which you are getting tests run, then you want to run them all. But what happened is, after the cold gets reviewed and merged, then another set of the same tests will come in. So, it is really important that you run all the tests before the code review and spend one hour, or are you okay with 80% of the confidence that happens in, like, five minutes, right? Then knowing that the rest of the 20% get code immediately after.

    I think there is an argument to be made that, overall, you’d be better off, even if some program slips down into the ____, so long as it gets eventually caught before the customer sees everything, so.

    Shimel: So, I guess then that that begs the question is—can that be accomplished? And, to accomplish that, what’s it going to take?

    Kawaguchi: Mm-hmm, yeah, and that’s what we are working on. I think, you know, the—so, I have some preliminary numbers, and they look good. Because if you think about it, the baseline that we are competing against is basically people running all the tests in a random order, and so, literally, you cannot do any worse than that. So, and instinctively as a developer, I know that certain tests are designed to catch certain programs in some specific things.

    So, really, the only question is how well a machine learning model can learn these things over what amount of data. And that, over time, I think I’m sure is gonna get better and whatnot. And I don’t know exactly what kind of impact we can create or what product, but overall, I’m feeling pretty confident. You know, it just needs to be good enough to start making an impact, and it’s only gonna go up from there.

    Shimel: Harpreet, what does success look like for you?

    Singh: Well, this has been a question that’s been asked to us while we were raising funds. I like to think in terms of—so, we’ve done this before. We’ve helped companies get successful. What’s in it for me is actually solving a problem for customers.

    So, if you are able to get and solve a software team’s problems everywhere, to get their products in the hands of customers first—that’s success for me. I know it’s not sort of putting the typical Silicon Valley 100 million to 200 million ARRs. That’s not where we are. That’s not where my head is at. It’s really helping people who are struggling with delivering software.

    Shimel: Got it. K.K.?

    Kawaguchi: So, you know—

    Shimel: What about you?

    Kawaguchi: One time I was working in San Francisco Airport, and then somebody was, you know, working behind me and she said, “Oh, you got a Jenkins thing. That’s nice. Where can I get it?” Right? And that, to me, is like a success. So, it’s gonna be a difficult one to beat once more, but you know, I feel like it’s worth giving it a shot.

    I mean, that’s the kind of—I guess I sort of think that the sign that you made a big enough impact in the world, and some people, and touched some people in pretty deep ways. And that, to me, is success. That’s what I’m itching for, yeah.

    Shimel: Okay. Let’s—let me hold both of your feet to the fire a little bit. So, you’re now launched, this is—you know, it’s out there, the website’s out at Launchable Inc, that’s L-A-U-N-C-H-A-B-L-E-I-N-C dot com, right? When can people hope, maybe, to see a beta? When do you think you’ll be out with something people can actually start using?

    Singh: You’re really holding our feet to the fire, right?

    Kawaguchi: [Laughter]

    Singh: You have a product manager and an engineer, here. I would like my engineering head to give me the dates on deliverables. [Laughter]

    Shimel: You know what? Classic. You’ll make a [Cross talk].

    Singh: Boy, you and me on the same side, here.

    Shimel: That was a great CEO move right there.

    Singh: [Laughter]

    Shimel: Alright, Mr. co-CEO, he threw you under the bus. What do you think?

    Singh: Jokes apart, I think one of the things that we are actually doing at the moment are, we are actively talking to prospects and customers. A lesson that we’ve learned over the years is not build a product in a vacuum and then hope that we’ve solved the problem.

    So, we are actually—there is actually a beta invite list button on our website. What we are looking for is actually partners who can come with us and tell us if we are solving the right problem and work with us to solve the right problem, right? So, that’s sort of where we are today. I’ll leave it at that and I’ll put the dates to you.

    Kawaguchi: Yeah, so, you know, we have preliminary numbers, like I mentioned. We made some models that made some impact, and now what I’m really hoping to find is event partners where we can apply the same technique on their system, their projects, and their source code to see what kind of impact does it produce. And then, so that’s—you know, that may be fundamental in these teams who are willing to work closely with us, and—

    Singh: Is that some number that you can share on preliminary numbers?

    Kawaguchi: Yeah. So, I mean, there is no guarantee that it’s gonna apply to your system. But, for example, while I was able to do this, select 10% of the subset for the best case use and it’s gonna catch like 40% of ____ right off the bat. So, I guess, you know, that’s—yeah, that, to me, is a pretty promising number, and you can only go up from here, so.

    Shimel: Excellent. So, look, both of you have garnered a ton of good will, not only among your peers as evidenced by the angel, you know, the investors’ list here, but also in the community at large, and K.K.—I mean, it goes without saying within the Jenkins community how well respected and, I don’t wanna use the term worshiped, that’ll go to your head, [Laughter] but you know, how honored you are in the community.

    What, people maybe from the community listening, what can they do to help, here?

    Kawaguchi: Yeah, I think giving us their feedback probably is the most. You know, we are working on something that we think is pretty exciting, and then we wanna hear that, or if that’s not the case, then that’s also great to hear. So, so far, the conversations we’ve had with people who are interested in this space who are thinking about this space have been incredibly valuable.

    So, if—you know, if you feel like some of what we discussed here is you and your challenges, we’d love to hear from you. And also, we are hiring—wink, wink. So, you know. [Laughter]

    Shimel: Everyone’s hiring.

    Kawaguchi: Yeah, so if you—

    Shimel: Alright. Is it on the website?

    Kawaguchi: It is. You know, one thing I got a lot more passionate at, you know, CloudBees and Jenkins is sort of passing back onto the next people, right? I felt like I got a lot more joy out of talking to the junior engineers and folks, we steer and mentor them. So, I feel like what we can offer—and I know it’s true with Harpreet, too, so I feel like what we have right here is an incredible opportunity to work, you know, sit side by side with these people, you know, people like us who, I think ____ creativity. We’re also passionate about mentoring other people. So, you know, I think if it sounds like your cup of tea, then…

    Singh: I’ll jump on that. So, I think, CloudBees was a fantastic company. You know Sacha, Alan—he’s just a great leader. He built a fantastic company. So, we learned a lot of good lessons on what it means to be a good company.

    Then I ended up at Atlassian where, you know, I learned the same things from Scott Farquhar and the other folks, and we’ve tried to bring in all the great points from these two companies into this company, right? We are sort of values driven. We’ve really thought hard about the kind of company we want to build, the kind of impact we want to have in the world, and we are looking for people who share that sense of mission, right, to come on our team, right? And if you find some of those guys on our team, that’s fantastic.

    On the former side of the house, again, we are looking to solve problems for customers. We are not looking to do this in a vacuum. So, there’s a general profile of people who are like, “Yeah, I hear this. I wish I could partner with someone and help you define what—go solve this problem for me.” If you’re that person, we would love to talk to you and we’d love to partner with you through those things.

    Shimel: Fantastic. Guys, we’re way over time. I promised you 15 minutes. We might be coming up on a half hour; I apologize.

    But let me close off by saying—you know what? Congratulations. Someone once told me new companies are like boats—the best days are the first day you get ‘em and the day you sell ‘em. [Laughter] But that being said, it’s also a lot of fun sailing ‘em. So, enjoy the sail, guys—change the world. If there’s anything, of course, we can do here at MediaOps, you know you can count on it. I couldn’t think of two nicer people for this new adventure and to take on this challenge.

    Harpreet Singh, K.K.—co-founders, co-CEO of Launchable, you can find out more at Launchable Inc. This is Alan Shimel for devopsy.kinsta.cloud. You’ve just listened to another DevOps Chat.

    — Alan Shimel

  • Five Challenges of Machine Learning DevOps

    Five Challenges of Machine Learning DevOps

    As organizations add machine learning (ML) to their workflows, it’s tempting to try to squeeze model creation and deployment into the existing software development lifecycle (SDLC). However, ML is fundamentally different than traditional applications, and it’s important to account for that in a new, unique process called the machine learning development lifecycle.

    We have identified five challenges every organization should keep in mind as they begin to support ML development.

    Making Heterogeneity Work

    Machine learning is successful when the right tool is selected for a given job. Depending on the use case, a data scientist might choose Python, R, Scala or another language to build one model, and another language for a second model. Within a given programming language, there are numerous frameworks and toolkits available, adding complexity to versioning and consistency.

    For example, TensorFlow, PyTorch and Scikit-learn all work with Python, but each is tuned for specific types of operations, and each outputs a slightly different type of model.

    ML-enabled applications typically call on a pipeline of interconnected models, often written by different teams using different languages and frameworks.

    Figure 1: An ML pipeline that extracts text from scanned documents and analyzes its sentiment passing through several languages and frameworks.

    For example, a public relations firm looking to identify news and reports critical of one of its customers might use the following pipeline:

    1. Extract all text from hundreds of scanned documents with an Optical Character Recognition (OCR) model.
    2. Identify the languages of that text with a language-identification model.
    3. Translate the text to English.
    4. Prepare the text for sentiment analysis with a vectorization model that turns English text to numbers.
    5. Use a sentiment analysis model to score the text.

    Planning for ML Iterating

    In machine learning, your code is only part of a larger ecosystem—the interaction of models with live, often unstructured or unpredictable, data. Interactions with new data can introduce model drift and affect accuracy, requiring constant model evaluating, tuning and retraining.

    Figure 2: The continuous ML Lifecycle with iterations.

    As a result, ML iterations are faster and more frequent than traditional app development workflows. This places greater demands on versioning and other operational processes and requires a greater degree of agility from DevOps processes, tools and staff.

    Organizations interested in adopting ML-enabled applications need to be prepared to handle the burden of iteration without breaking the budget or introducing needless infrastructure tasks.

    Developing the Right Infrastructure

    The model training process typically involves multiple instances of the following:

    • A long and intensive compute cycle.
    • A fixed, inelastic load.
    • A single user.
    • Concurrent experiments on a single model.

    After deployment, models from multiple teams enter a shared production phase characterized by:

    • Short, unpredictable compute bursts.
    • Elastic scaling.
    • Many users consuming many models.

    Operations teams must be able to support both the training and deployment environments on an ongoing basis, because it’s a continuous lifecycle. Selecting the right infrastructure for that job is complicated by a rapidly evolving tech stack, from the increasing variety of processors available for ML workloads to advances in cloud-specific scaling and management, to a never-ending stream of new data science tooling.

    It’s important to future-proof your ML program to ensure a tool or framework you use today does not lead to lock-in or worse—obsolescence—down the road.

    How Will You Scale?

    To address the unpredictability of ML workloads and the premium on low latency, organizations must build compute capacity to support substantial bursts.

    There are three typical architectural approaches, each with benefits and drawbacks:

    Traditional Capacity Planning: In a traditional architecture, operations reserve compute resources capable of scaling to maximum anticipated demand.

     

    Pros:

    • Reserved capacity is always available
    • Easy to administer

     

    Cons:

    • Extreme waste and cost
    • Unanticipated demand can exceed the capacity

     

    Elastic Scaling: Standard elastic scaling designs for a local maximum, scaling machines up and down based on step functions.

     

    Pros:

    • Cost improvements over traditional architecture

     

    Cons:

    • Inefficient hardware utilization
    • Difficult to manage heterogeneous workloads on the same hardware
    • Slightly more management overhead than traditional architecture

    Serverless: Serverless architecture approaches spin up models as requests come in.

     

    Pros:

    • Significantly improved hardware utilization
    • Dramatically lower costs
    • Easier to run and scale different types of workloads on the same machines

    Cons:

    • Substantial administrative overhead for custom-built, on-prem solutions
    • Some third-party services without extensive APIs and reporting provide limited explainability
    • Performance tuning to meet latency

    Architects can implement any of the three approaches in the cloud or in a physical datacenter, though elastic scaling or serverless approaches requires far more effort to manage in-house.

    Trusting the Results with Auditability and Governance

    Explainability—understanding why models make given predictions or classifications—is an essential part of ML infrastructure and the lifecycle. Equally important and often overlooked, however, are the related topics of model auditability and governance—understanding and managing access to models, data and related assets.

    Figure 3: An audit trail can help inform decisions.

    To make sense of complex pipelines, a multitude of users and rapid model and data iteration, deployment systems should attempt to identify:

    • Who called which version of which model.
    • At what time a model was called.
    • Which data the model used.
    • What result was produced.

    Thinking about and planning for these five challenges before beginning a machine learning initiative is a good place to start to ensure a valuable return from your investment.

    To learn more about containerized infrastructure and cloud native technologies, consider coming to KubeCon + CloudNativeCon NA, November 18-21 in San Diego.

    — Diego Oppenheimer