devopsy.kinsta.cloud

Category: DevOps Practice

Elevating Quality Engineers in an AI-Driven Landscape

Artificial intelligence (AI) is quickly transforming software testing and quality assurance (QA) by powering game-changing capabilities like autonomous testing, intelligent test case generation and real-time risk analysis. As a result, it is creating new opportunities for AI-savvy test engineers. But that doesn’t mean that you can set it and forget it.

AI helps us work smarter by eliminating repetitive tasks. Think of it like a supercharged intern who can rapidly analyze huge datasets, suggest myriad test ideas and point out risks — all at incredible speed and scale. But also, like most interns, AI needs some guidance. With the addition of thoughtful human oversight and control, AI tools can ultimately enhance the work of test engineers, and even alleviate stress by automating mundane work. In the end, however, human test engineers will remain the key decision-makers.

Buckle Up: The Landscape is About to Change

AI enables QA professionals to focus more time and energy on strategic, high-value testing activities by automatically handling rote script maintenance, data management and other repetitive tasks that consume excessive time and effort. It can eliminate the grunt work so test engineers can focus on more rewarding testing aspects of their jobs. AI also reduces over-reliance on specialized technical expertise by shifting the focus back to foundational testing skills like business domain knowledge, advanced risk analysis and cross-team collaboration. Ultimately, a test engineer’s true value is a combination of their experience and their domain knowledge, not just their tech skills. And AI can help bring these to the forefront.

As AI helps developers produce more code faster, it will create more work for quality assurance testers. Using AI to automate their test plans, test engineers will be able to manage greater volumes of throughput without feeling overwhelmed by it. To be sure, this will all be a boon to the software industry as a whole. So, buckle up and prepare for more — not less — work.

Broad adoption of AI will organically transform QA roles into more advisory-focused, strategic positions focused on optimizing end-to-end testing processes versus hands-on test execution. Test engineers will transition from being “tactical doers” to a role akin to a strategic coach who guides the technology for maximal impact.

AI-enabled autonomous testing tools that automatically generate test cases, execute scripts, manage test data and identify risks will become standard components of future QA stacks, allowing test engineers to achieve much higher test coverage rather than simply automating happy path testing. But this can only happen if those in control ask the right questions, establish sound guardrails and apply the unique skills only a human can provide.

The Importance of Human Judgment — Don’t Relinquish Control

The key to the successful integration of AI into your testing program is to remain skeptical. It is unwise to blindly trust in the opaque limitations of an AI tool. It also helps to know how the technology works under the hood. In fact, irresponsible use of AI can be dangerous, especially if we don’t understand its limitations.

Testers need consistent human judgment and involvement. Human skills like customer empathy, communication and creative problem-solving remain critical, and AI will never replicate these distinctly human capabilities. By emphasizing their focus on users, systems and applications, creative testers will thrive in the presence of AI automation.

It is not only software testers who need to embrace change but developers of testing tools, as well. It is important that whoever creates the platforms for testing provides transparency into how their tools work. AI tool developers should avoid “black box” development and provide visibility into their models. When a test engineer understands how an AI tool works under the hood — and how it is limited — they will be in a better position to trust and verify the tool’s output. This open collaboration between tool developers and test engineers can maximize benefits and minimize risks in our industry.

Preparing for an AI-Driven Future

To best prepare for the adoption of AI-powered QA tasks, enterprises must invest in proper change management, provide extensive AI training and promote open communication. For their part, test engineers can benefit from engaging with each other to share their AI experiences — whether positive or negative.

In the short term, the software test industry needs leaders to take a measured approach by piloting AI tools on targeted use cases, versus attempting to overhaul all of testing overnight. Start small with low-risk AI projects, then scale what works.

While AI promises to change the testing status quo forever, human insight will remain the indispensable ingredient for responsible, successful software releases. Creative testers who master AI will drive the greatest innovations. Those who embrace this exciting time in our industry are sure to come out ahead of the pack.

November 14, 2023
Elastic Previews Unified Query Language for Search Platform

Elastic this week previewed a standard query language that can be used across its portfolio to streamline investigations into IT and cybersecurity events.

Gagan Singh, vice president of product marking for Elastic, said Elasticsearch Query Language (ES|QL) makes it simpler, via a single query, to aggregate data from multiple sources using a piped query syntax that makes it simpler to iteratively interrogate data.

Scheduled to become generally available in 2024, ES|QL replaces the multiple domain-specific languages the company has been providing for each version of its search engine. Instead of having to move data into a central repository, Elastic Seach makes it possible to query data wherever it resides in an organization, noted Singh.

That approach enables, for example, a site reliability engineer (SRE) to discover the root cause of an issue through a series of queries that can be launched from a familiar search interface, he said. It will also foster increased collaboration across IT teams that need to visualize query results, added Singh.

In general, IT teams are struggling with an exponential increase in the amount of data they need to analyze to discover the root chaise of an issue, noted Singh. The Elastic search engine provides a means to analyze that data at the level of scale that is now required, he added.

The preview of ES|QL comes on the heels of a technical preview of a generative artificial intelligence (AI) tool for observability the company has made available. The Elastic AI Assistant for Observability makes use of large language models (LLMs) and the Elasticsearch Relevance Engine (ESRE) to explain application errors, log messages, alerts and recommendations for improving code in plain language. DevOps teams can visualize and interactively chat with telemetry data and then invoke runbooks to remediate issues.

Elastic has also generally made available a Universal Profiling tool to enable DevOps teams to identify bottlenecks that adversely impact code performance.

There is little doubt that fundamental changes to how DevOps and cybersecurity teams interact with telemetry data are on the way. The challenge now will be determining when to rely on a generative AI tool versus diving deeper into that data by launching a series of queries. At the very least, AI should provide DevOps and cybersecurity professionals with a better sense of where to start an investigation.

Ultimately, the goal should be to reduce the level of DevOps and cybersecurity effort that is currently required to succeed. In theory, as it becomes simpler to investigate issues, the number of organizations capable of embracing DevOps best practices should increase as well. That’s crucial because IT environments are only becoming more complex with each passing day. The number of dependencies that exist between services has become too numerous for IT teams to track manually. More challenging still, those services are dynamically being updated, so what was known today about them might no longer be relevant tomorrow.

Hopefully, the amount of time and toil required to address issues will significantly decline in the months ahead as the next generation of tools becomes available. The challenge now is hanging on long enough for them to be battle-tested.

November 10, 2023
Application Modernization: A Catalyst for Fixing Sub-Optimal UX

Today, some of the biggest government programs in existence are Medicare and Medicaid with the two health care insurance providers serving over 43% of the U.S. population. The Centers for Medicare and Medicaid (CMS) spent $1.6 trillion in 2021 and continue to grow in both size and cost on an annual basis. With this type of investment and permeation into the American populace, one would assume that this program would be running on the most up-to-date applications available. However, both public and private health care insurance carriers continue to operate on legacy systems.

Recently, Andrea Fletcher, the digital services director of the CMS, described their system as operating on technology from the 1960s, which is alarming considering the responsibilities of the program. In fact, the Biden administration has only recently announced plans to modernize their organ matching system. This announcement came from a study conducted in February, which showed that only 26% of organs were procured from donors on average. This is just one example of how efforts to improve legacy systems, which are currently being used by over 40% of the population, allow both public and private organizations to operate in a more efficient manner as well as satisfy user needs. Further exploration will show how modernization improves other areas of business, such as creating user experiences that are both easier to navigate and more personalized as well as the ability to incorporate the latest cybersecurity updates.

Modernizing as a Means to Improve User Accessibility and Personalization

Given the size and diversity of the United States population, a modern flexible system is necessary for creating a smooth user experience for every individual. Today, users connect with insurance carriers using a variety of different devices while communicating using different languages and possessing different levels of technical expertise. The ability to create a personalized user experience for each individual will require a level of modernization that legacy systems were not built for. The importance of this flexibility is accentuated when considering that the purpose of Medicare and Medicaid, in particular, is to serve the elderly and underprivileged.

Prioritizing the modernization of the consumer experience is not limited to just government organizations. In an independent survey conducted by Evolveware, 46% of private organizations measure the success of their application modernization efforts through an expected improvement in customer experience. As health care insurers continue to rely more heavily on analytics and the population continues to adopt digital technologies, the process of creating a streamlined and targeted approach to purchasing and managing health care becomes essential to staying competitive. For a multitude of companies in the marketplace, having an agile system that can be customized to each individual user can be a major differentiator, especially given the variety of factors that each individual has when it comes to making decisions about their health. Moreover, by prioritizing the user experience, organizations can improve their brand loyalty and customer retention, which are key factors when dealing with a competitive marketplace with a lot of players. Insurers have recognized this, as a recent survey of CIOs by Gartner showed that improving the customer experience and application modernization were listed as the highest priorities in the insurance industry. As customers become more judicious with their money due to current macroeconomic trends, providers will need to make sure their platforms’ user experiences are on par with their competitors.

Modernization to Mitigate the Impact of Hidden Costs and Security Vulnerabilities

The costs of not updating legacy systems should also be taken into consideration when making the decision to modernize. While the expense and process of moving away from legacy systems can seem like a significant obstacle, these systems, due to on-site maintenance and operational costs, are already expensive to maintain on a continuous basis. In fact, health care IT teams spend around 60% of their time trying to maintain and manage legacy systems. Furthermore, the opportunity cost of IT teams focusing on maintenance over innovation is significant, especially given the potential of the latest technologies to deliver a more intuitive user experience. There is also an understated but pressing labor cost with legacy systems, as members of the workforce who are familiar with these systems are aging and retiring, and the replacement workforce is not well-versed in legacy languages and technologies.

Finally, legacy systems can also contain blind spots that are prime targets for cybersecurity breaches and hacks today. Legacy applications were built at a time when these types of attacks were not common and, therefore, don’t have the most up-to-date security capabilities embedded. Many of these systems also do not have up-to-date documentation, resulting in a high potential for hidden vulnerabilities. For health care organizations that have access to sensitive data, understanding and then shoring up these vulnerabilities using modern security features is critical.

Health Care Modernization Example: Chatbots

One prominent example of how application modernization can lead to organizational success is the integration of chatbots. As technology expands and businesses identify new ways to reach consumers, the addition of chatbots has become more prominent on platforms across industries. For the health care insurance industry, the bots provide both a cost-efficient way to process insurance claims as well as an enhanced user experience through ease of use. Recent estimations show that bots can handle the workload of two to five claims processors and can deliver 20 times the production relative to cost, allowing industry employees to focus on less tedious and more innovative tasks. Using generative AI models, chatbots will continue to become more sophisticated and varied in their responses to user input. As a result, modernizing legacy systems to optimize the integration of these bots becomes integral to remaining competitive in the marketplace.

Legacy Systems and the Future

As we look to the future, both government and private insurance carriers have plenty of internal and external factors that incentivize application modernization. As Medicare and Medicaid continue to grow due to the increasing number of eligible users, an agile system is integral for user accessibility. Meanwhile, macroeconomic factors such as high costs and an increasingly competitive marketplace will force private organizations to adopt systems that can integrate and adapt to the newest technologies for a better and more secure customer experience. With serious challenges facing both the public and private sectors of healthcare insurance, the necessity to modernize has never been greater.

November 10, 2023
Boost User Experience With Global Network Monitoring
With the increasing global use of apps and services, ensuring a smooth user experience across different geographic locations is already a top priority for many DevOps teams. However, traditional setups, such as maintaining servers in multiple locations, can be a costly and inflexible approach, especially for smaller DevOps teams or those with tight budgets.

In this article, we’ll explore how DevOps teams of all sizes can leverage global network testing platforms to set up simple and flexible global monitoring that fits well with existing workflows without breaking the bank.

Operating on a Global Scale Requires Flexibility

To monitor apps and services from various locations around the globe, teams need access to a globally distributed infrastructure.
Typically, DevOps teams consider different setups to meet their needs, including:
- On-premises servers: Larger companies with data centers or offices worldwide can use these locations to set up network monitoring and troubleshooting.
- Cloud service providers: Infrastructure-as-a-service (IaaS) solutions allow teams to deploy virtual servers to multiple locations without needing physical access to those sites.
- Third-party services: Feature-rich monitoring tools with servers in multiple regions enable teams to leverage existing infrastructure that they don’t have to maintain themselves.
While these setups offer various benefits to different project and team requirements, they also come with drawbacks that can impact a DevOps team’s efforts:
- Cost: Infrastructure, whether self-managed or acquired via third-party vendors, can be a significant expense – especially when teams need to monitor and test from multiple locations. Moreover, third-party monitoring services may come with unneeded features, resulting in unnecessary costs.
- Flexibility: Most setups don’t allow users to change the location from which to test or monitor as needed, making it impossible to work on a real global scale. Plus, they can often not cover all the areas from which teams want to make measurements.
- Complexity: Adding new hardware, cloud instances and tools to workflows introduces new complexity and challenges, especially for DevOps teams that already struggle to complete tasks efficiently.
To address these issues, teams should consider deploying a global network testing platform that allows them to run network tests globally and overcome the challenges described.

Leveraging Global Network Testing Platforms

Global network testing platforms such as Globalping or RIPE Atlas provide a reliable solution for effortlessly performing network measurements from any location worldwide. Note, however, that these platforms are quite different from extensive continuous monitoring platforms like Kentik. They are much more streamlined and focus on delivering on-demand network testing capabilities through APIs, CLIs and other integrations.

But how do these platforms enable teams to run network tests from anywhere? They do this by leveraging a dense network of thousands of probes hosted by partners and individuals worldwide. These probes serve as the hosts from which network commands, such as ping, traceroute, mtr or dig, are executed. As a result, platform users can test and examine various aspects of their website, CDN or DNS server – even in the far corners of the world.

This image shows an example of the Globalping CLI displaying ping results to devopsy.kinsta.cloud from probes in Berlin and South America:

Transform On-Demand Testing Into Monitoring

While on-demand testing and monitoring are not exactly the same, global network testing platforms can still serve most DevOps teams’ monitoring setups.

These platforms usually provide APIs and CLI tools that let users make measurement requests programmatically so they can use these integrations for their own purposes.

Here are two practical and straightforward use cases to turn on-demand testing into monitoring:
- Teams can create customized global monitoring tools using the platform’s API or extend their existing setups. They can make measurement requests from various locations, collect the data and use it to set up alerts and draw graphs.
- For a simple and lightweight monitoring solution, DevOps engineers can write a Bash script that automatically makes measurement requests and sends emails if results are not as expected.
Choosing the Right Tool

Want to try network testing platforms but don’t know how to choose the right one for your team? Here’s what to consider:
- Integrations: What integrations does the platform offer? APIs and CLI tools are essential, allowing you to create custom implementations and integrate them with existing tools. Also, look for platforms with user-friendly web-based interfaces that enable non-technical users to run tests.
- Collaboration features: Consider platforms that support collaborative network testing, which is especially important for distributed DevOps teams. For example, Globalping offers a Slack app that brings team members together in a virtual space to solve problems faster.
- Network density: Choose a platform with a dense probe network to ensure reliable measurement results 24/7. You may also want to learn more about the probes themselves. For example, you can review the code to see whether it meets your security requirements if the project is open source.
- Rates & limits: Check the pricing structure and limits carefully, as platforms take different approaches (especially if you need to run custom measurements). Globalping, for example, is a free, open source platform with generous limits that you can extend by becoming a GitHub sponsor. On the other hand, Ripe Atlas uses a “credits” system where you can buy credits or earn them by hosting probes.
- Community & Support: Look for a platform with an active community that supports each other in forums and chat groups. Also, explore the platform’s documentation, tutorials and support channels available to help you get started and assist you in resolving issues along the way.
In summary, choosing a global network testing platform enables DevOps teams to monitor and troubleshoot their infrastructure from anywhere in the world flexibly and cost-effectively. By leveraging the capabilities of these platforms, teams can achieve higher service reliability and a better user experience, even in the most remote corners of the world.
November 10, 2023
Survey Surfaces Software Supply Chain Security Challenges

A survey of 268 security leaders and 252 developers published this week found that while there is general agreement on the importance of software supply chain security, there is a significant disconnect on what is required to achieve that goal.

Conducted by the Harris Poll on behalf of Chainguard, a provider of curated container images for building secure software, the survey found 92% of developers acknowledged software supply chain security is at least very important, with 39% noting it as absolutely essential. Similarly, 96% said effective software security practices are important to meeting government or regulatory requirements, with 93% noting effective software security as a critical component of their threat and risk mitigation strategy.

Overall, most security leaders (70%) and CISOs viewed software supply chain security as a top priority. However, only slightly more than half of developers (52%) agreed. Nearly three-quarters of developers (72%) said they are very security-conscious, but only 50% of security leaders rated developers as very security-conscious.

Roughly two-thirds of security leaders (69%) and developers (64%) agreed that lack of communication and collaboration between developers and security teams is a problem. More than three-quarters of security leaders (77%) and more than two-thirds of developers (68%) agreed that the need to prioritize security causes tension between their teams. Nearly three-quarters of developers said the work/tools their security team requires them to use interferes with their productivity and innovation. Tools in place included software bills of materials (SBOMs) (40%), and nearly half are implementing software supply chain security frameworks such as Supply-chain Levels for Software Artifacts (SLSA) (47%) and the Secure Software Development Framework (47%) defined by the National Institute of Standards and Technology (NIST).

The report also found that 36% of security leaders and 34% of developers viewed an overwhelming number of scanner false-positive vulnerability alerts as being among the biggest software supply security obstacles an organization faces.

Most security leaders (74%) and developers (85%) believed that prioritization of software supply chain security would increase over the next five years.

However, many developers don’t believe enough security teams understand core software. Developers said security teams are generally familiar with open source software libraries and projects (61%), source code repositories and source code management systems (60%) and software build tools (59%), but only 43% said security teams are very familiar with container images.

On the plus side, many security leaders and developers understand that best practices and tooling in software security to drive certain business outcomes is essential, including customer retention (43% and 40%), meeting or satisfying procurement contract obligations (36% and 32%), fewer breaches or compromises (34% each), and developer/engineer productivity (32% and 34%).

Kim Lewandowski, chief product officer at Chainguard, said the survey makes it apparent that while progress toward securing software supply chains is being made, much work still needs to be done as silos between the developers and security leaders who are ultimately responsible for application security persist. While more organizations than ever have embraced DevSecOps best practices to achieve that goal, the cultural and technical challenges many organizations encounter are often underestimated, she noted.

The issue, as always, is that given the limited resources available, many organizations, despite their best efforts, are clearly still finding it difficult to stay current with updates to remediate vulnerabilities as they become available.

November 9, 2023
Salesforce Winter Release: What you Need to Know

With the Salesforce Winter release dates behind us, it’s time to evaluate all of the new innovations added to your Salesforce ecosystem. Let’s just say that many users will be excited about the features added to this release cycle; plus, there were many anticipated updates that have finally come to fruition.

Let’s dive right in and get to the meat of the release that will appease the administrator community, automate internal processes and close many operation gaps, and promote more team collaborations for you and your clients.

Search Capabilities and Admin Processes

These updates make an admin’s day-to-day work more streamlined, automated and convenient when it comes to getting things done.

● Configure Searchable Objects: This update allows admins to determine searchable objects available by Profile. This will allow users to broaden their search capabilities of records and more quickly and easily find what they need. For example, you can search for a picklist value and see all the records with that chosen value. Users, especially those in larger organizations, have been wanting this as it will streamline their process for customers. *This is in beta.
● Refine Your Search Results With Sort: Another win in the Search department – now you can sort your searches by different attributes and column headers. A nice “sigh of relief” feature for users, allowing them to find what they need faster. *Created because of users’ input on IdeaExchange.
● Dynamic Forms on Standard Objects: Administrators will very much appreciate this update as it streamlines the admin process with enhanced page performance and optional visibility rules. This will be rolled out everywhere except in three areas: Campaigns, products and tasks (though we really look forward to when this will be available here.) *Created because of users’ input on IdeaExchange.

AI

Since Einstein was introduced to the Salesforce community, it has helped users work smarter and faster, which is why seeing some of these key updates come to light, as noted below, has been a game-changer for many.

● Ability to Report on Emails Captured by Einstein Activity Capture (EAC): When EAC was first introduced, it was a huge weight off of admins because there was no more logging and it helped synchronize all conversations in one place. Over time, you couldn’t access or report on the data because it lived outside of SF, so you couldn’t compare it to other data in Salesforce, i.e., how many conversations does it take to close a deal? However, with this new update, you can! This is an enormous feature gap that has been filled, and will ultimately save people time and money as many would outsource this work to third-party vendors. *Created because of users’ input on IdeaExchange.● Call Summaries Powered by Einstein: This is fully available, and just a generally cool feature where Einstein will record voice and video calls and transform them into summaries that are editable and include next steps and customer feedback.
● Conversation Sentiment Analysis: This feature goes hand in hand with the call summaries because now, customer success agents can listen and view sentiments of a conversation between agent and customer, distinguishing if they are happy, calm, disgruntled, etc. There’s a sentiment score and visuals provided for easy reporting as well. This is a great feature for agent performance grading and identifying coaching opportunities.

Flows

Flows continues to be a priority and focus for Salesforce, and they certainly delivered on giving users what they want with these improvements.

● Screen Flows With Reactive Components: Users and developers will very much appreciate this update as it reduces the number of screens they have to click through. With reactive components, the user interface can react to the user and what it puts on the screen. Previously, in order to use the choice a user selected, you’d have to click “next” for it to recognize and load the next screen. This could turn into three to four “next” clicks to get where you need. Now, the choice can be calculated on the fly and the dependent choice will appear below, without the need for extra click-throughs!
● Custom Error Messages in Record-Trigger Flows: What was once an automated feature, now puts the user in the driver seat. With a custom error message, you can now create targeted – and user-friendly – error messages for your customers and explain what went wrong and/or how to correct it. This is a big win in our opinion, which will not only save the end user headaches and blockers but will also save your admin or consultant time researching where and what happened. This is now generally available – during the Summer release, this was still in beta. *Created because of users’ input on IdeaExchange.
● Autolaunched Flows Based on Data Changes in Data Cloud: Based on a data model object or calculated insight object, you can start a data cloud-triggered flow. This allows you to apply flows to your data cloud, storing all of your business data, and of course Salesforce data too.

Callouts

It wouldn’t be fair to close out the review of this release update before highlighting a few noteworthy callouts that will surely resonate with certain users, which include:

● Automated Site Check-Ins and Check-Outs: For mobile workers, this new feature protects their privacy while also tracking the times they arrive and leave job sites. It provides automated timestamps for location-based actions and gives you a report based on those insights. This is being piloted so not fully available yet.
● Controlled Tabs and Sub-Tabs: This change is currently in beta, but will resonate with the day-to-day user as it makes it immensely easier to manage these tabs with the methods they provide.

There’s a lot to experience and discover within the Winter ‘24 release, from increased workflow to automated operations; any user will be happy. But Salesforce also delivered for its users. Many of the ideas posted on IdeaExchange are now reflected in these new features, which just goes to show that Salesforce is listening. Let’s continue using these innovations and work with Salesforce to continue bettering the overall experience for us and our customers. We can only imagine what Salesforce will conquer with the Spring ‘24 release!

November 9, 2023
How Open Source Software Powers High-Impact Climate Science

According to the NOAA National Centers for Environmental Information (NCEI), the past nine years have been the nine warmest since record-keeping began in 1880, and 2023 is on track to set another record. Making the hard decisions involved in addressing this global crisis requires analyzing and understanding enormous quantities of data using sophisticated computational methods. Luckily, innovations from the open source software community are helping revolutionize climate science, letting researchers quickly derive insights from data.

Delivering Critical Collaboration

Modern science relies heavily on complex computational and numerical analysis techniques. However, the massive amount of measurement and simulation data available in fields like climate science makes traditional approaches impractical. With petabyte-scale data, researchers can no longer expect to download or obtain physical copies of entire datasets, nor can their local workstations or laptops handle datasets of that size once they arrive. Moving the scientists to the data is also impractical since researchers all over the world need to make use of the same data.

Instead, scientists and engineers from Pangeo.io and other global collaborations have developed a robust set of open-source tools for working efficiently with large, remote datasets, bypassing restrictions from proprietary software that’s limited to small datasets installed locally. Instead of monolithic commercial applications or inflexible Fortran codebases, modern researchers use flexible and lightweight approaches for distributing computation across cloud computing resources shared between many collaborators. These innovations are critical for climate science since further delay could lead to catastrophic challenges in combating global warming.

Deriving Faster, Flexible Data for Greater Output at Scale

Python’s scientific ecosystem, in particular, has gained prominence and is top of mind when considering open-source platforms best suited to execute climate solutions. Incoming graduate students more often know Python than any other programming language now that Python has taken over as the world’s most popular programming language. Python’s open-source ecosystem includes a wide variety of packages, each focusing on a specific area of functionality. These components evolve quickly and regularly introduce new features while maintaining compatibility through standard protocols. Unlike a monolithic, centrally designed approach, Python’s open ecosystem provides more flexibility when scaling data and solutions.

In high-impact fields like climate science, the goal is to enable scientists and researchers to concentrate on their domain-specific questions rather than becoming entangled in the intricacies of data manipulation and resource management. The numerical Python stack of open-source tools, combined with the ability to access available data in the cloud on a massive scale, facilitates this division of labor. Scientists can focus on writing software for the analysis parts specific to their expertise, meaning faster progress toward climate solutions and products.

With modern cloud-computing approaches, it makes more sense to move computation to the data rather than vice versa (since code is much smaller and more portable than data). With a platform like Jupyter running on a remote cluster and remote Python kernels as the execution environment, scientists can leverage the same infrastructure that hosts massive datasets in cloud object stores. Improvements in software technology and data formats also make it simple to access specific chunks of very large files, such as the temperature records associated with one particular city out of the entire world’s data. Accessing data becomes fast, cost-effective and scalable, bypassing traditional workflows involving repeatedly downloading large datasets consisting primarily of data not needed for a particular analysis.

For climate science specifically, this flexible remote-computing approach also means easier collaboration between developers and scientists to create innovations that could move climate studies and solutions forward. Distributed computing technologies such as Dask, Ray and Spark enable parallel and out-of-core computing, allowing for processing larger datasets at higher speeds. Several remote Jupyter-based services now exist, enabling even the main Python process (the “kernel”) to run remotely while utilizing the browser as an interactive development platform. Because all these tools are open, researchers can apply emerging technologies such as machine learning (ML) and artificial intelligence (AI) techniques as soon as they are developed without waiting for software and hardware manufacturers to catch up.

Creating Data Visualizations and Dashboarding

For humans to understand and validate the data and the steps involved in analysis requires visualization at every stage, which is difficult when the data is too large to transfer locally for display. New open-source tools like Datashader and VegaFusion offer a server-side rendering of even the largest datasets, efficiently rendering distributed datasets into a viewable image and then transferring only that image to a scientist’s local browser for display. Matplotlib, HoloViews, Altair, Plotly and hvPlot in Python, along with Makie.jl for Julia, now allow server-side rendering to be embedded into fully interactive plots in a web browser, letting researchers explore datasets and identify interesting features or detect issues with data quality or analysis steps. The Panel and Dash dashboarding tools in Python allow these plots to be combined into complete web applications for researchers to share their interactive analyses, all without any proprietary tools or restrictions (see this USGS wind turbine application as an example).

In a field with such a dire need for fast innovation, open-source technology is crucial to providing the creativity and speed required for climate scientists to fight global warming. The collaborative and transparent delivery of innovation is what the scientific community is increasingly counting on.

Dr. Martin Durant, staff software engineer at Anaconda, contributed to this article.

November 9, 2023
Generative AI: Some Parts of the Future Are Cool

I’ve been around long enough that I normally roll my eyes when ‘the new shiny thing’ shows up. If I start gushing about something, I truly see it as game-changing in a market space that is full of things that simply are not. Perhaps an example is in order. For years after college, my hobby was compilers and linkers. Until my Master’s program had a class in compilers and the newer variants of Lex and YACC showed me that those skills are archaic. Lots of fun, and I’ve kept my source to a 100% original MS-DOS linker, thinking one day I’ll update it to generate modern executables. But I won’t because the other thing that reduced the appeal of compiler and linker skills is open source. I can download code that, by virtue of being exercised by like a billion people, is inarguably better than my efforts were. Better to put my attention elsewhere. Open source was indeed a game changer. Today, when I see there is a new language du jour, my very first question is, “Okay, but why?” the answer is rarely, “To make developers’ lives easier.”

The same is true with generative AI. Just as open source has spawned a never-ending collection of failed or misdirected OSS projects, AI is starting to create buzz about dozens of “that’s not really useful” use cases. But amongst all those are the “This changes everything” use cases, and for IT–be they Devops or devOps, generative AI for code creation is huge. This will almost immediately become a mandatory job skill, and you should start learning how to get generative AI to spit out the code you need today. It will increase your productivity, help your company, and it will increase your career prospects, helping you.

Generative AI Goes Further

Two examples. In completely different projects that aren’t even for companies in the same space, Lori and I both used generative AI to develop systems faster. I know very little about Lori’s work other than that it was to generate Python that did the grunt work and took care of the repetitive bits she hates. My work was to generate bash scripts for things I don’t generally do in Linux. Any honest assessment agrees that the power of Linux is masked by the insane level of obscure commands. While researching online has helped us get over the “What the heck does that do?” for decades, generative AI goes one step further. It allows you to tell it what you want to do, then give it whatever additional information it might need to tweak the results, and the script is done.

We all use APIs, libraries, SOs and snippets pretty much daily; this is literally just an extension of that usage. Most of us would search online for examples of what we needed to do, then copy and modify them. This really is literally, “Give me an example of that. Good, now customize it to my needs.”

There are risks with any significant change, and generative AI for script development is no different, but the largest risk identified – that entry-level people will not learn and grow – is blown out of proportion. Given an example to read, the curious and those who are going to be responsible for supporting the system will go over it with a fine-toothed comb. The learning will be more reading than doing at that point, but in the end, the script still has to execute. Troubleshooting won’t perform itself, and people will continue to strive to understand — both because they need to when supporting systems, and because curiosity is indeed the hallmark of IT staff.

You all are the backbone that keeps the organization online and employees productive. We are way past the time when 90% of companies would fail without IT. Keep rocking it, but give your org that competitive edge. Take any tool that will increase your productivity and help your career in the process.

November 8, 2023
IT Service Incidents Are Becoming More Frequent, Survey Says

A survey of 1,000 IT operations, DevOps, site reliability engineering (SRE) and platform engineering professionals in the U.S. conducted by Transposit, a provider of an incident management platform, found more than two-thirds (67%) have seen an increase in the frequency of service incidents that have affected their customers over the past 12 months.

Announced at the Kubecon + CloudNative conference today, the survey found 62% of respondents have also seen an increase in the amount of time it takes to resolve incidents over the course of the last year, with 80% reporting it takes up to six hours on average to resolve incidents from the first alert to mitigating the issue.

Paradoxically, however, 71% of respondents also claim to have automated incident management to their satisfaction, with 59% having a defined incident management process in place. One-third (33%) said only 11-25% of their incident management tasks or workflows are automated.

Transposit CEO Divanny Lamas said despite that level of comfort it’s clear the number of incidents that need to be addressed are continuing to increase as IT environments become more complex. In fact, in the next 12 months, 72% expected their organization to expand its IT stack.

At the same time, costs per incident are rising, with organizations at risk of losing up to $499,999 per hour on average, according to 63% of respondents. Nearly half (47%) said downtime can cost anywhere from $100,000 to $2 million.

A full 85% of respondents are hopeful that generative artificial intelligence (AI) will help further streamline incident management processes, with 80% having already embraced AI to varying degrees. More than half (51%) feel AI is making their job better, with 65% using it to improve the accuracy and quality of data. Just over half (51%) reported faster time to incident resolution, with 50% using it to more quickly and easily identify the root cause of issues, potential threats and vulnerabilities. Just under half (48%) use it to automate repetitive tasks or processes, streamlining their operations effectively.

A full 90% of respondents said integrating generative AI capabilities into incident management tools or platforms decreased the time it takes to create new automations.

Overall, 43% reported current incident management processes are not effective or are only being used by some team members due to confusing documentation (41%), limited access to tools (40%) and reliance on institutional knowledge (40%). Well over a third (37%) reported that only select team members have a comprehensive understanding of the defined incident management process and that they are adhered to consistently.

A full 96% said they believed it would be beneficial if all the tools their organization used to manage incidents were integrated through one tool or platform, the survey finds.

Top challenges cited included not enough buy-in from leadership or management (57%), not enough sharing of knowledge (54%), inadequate documentation of institutional knowledge and existing processes (54%) and a lack of clarity about what to automate (52%).

Respondents reported they expected to embrace technologies including AI or tools that employ machine learning algorithms (60%), automation tools or applications (53%) and communication/collaboration tools or applications (48%).

In addition, over the past year, 62% of respondents have increased their focus on SRE practices, with 58% planning to increase adoption of platform engineering as a methodology for centralizing the management of DevOps workflows.

Ultimately, automation should reduce incident management stress for all concerned, said Lamas. The challenge, as always, is making it simpler to achieve that goal in a way that doesn’t rely on a complex mix of custom scripts that don’t scale, she added.
One way or another, it’s clear that when it comes to incident management, there is still plenty of room for improvement.

November 7, 2023
The Benefits of Self-Service Infrastructure

Self-service infrastructure is a modern approach to IT management that enables development teams to take greater control of their systems and services without relying on dedicated IT staff for assistance. This approach offers numerous advantages that help businesses become more efficient and agile.

What is Self-Service Infrastructure?

Self-service infrastructure is a type of IT infrastructure system that allows users to access, manage and maintain their own applications, data and resources without requiring intervention from an external provider. It combines software automation tools, cloud computing services, open source technologies, network management solutions and other digital tools to provide an environment where users can interact with technology on their own terms, although within defined boundaries (created, i.e., for security reasons).

Self-service infrastructure has helped transform traditional service delivery models by increasing the development team’s autonomy when they use technology for various purposes. With self-service infrastructure in place, it can be easier for businesses to reduce costs associated with maintaining IT systems while also improving the speed at which operations can be performed since most tasks do not require manual intervention.

Some self-service infrastructure platforms include AWS Service Catalog, Google Cloud Deployment Manager, Spacelift and Backstage.

Self-service infrastructure comes with a bit of a challenge. There is often a learning curve associated with setting up an effective self-service architecture. Since software platforms are constantly evolving due to updates or new versions being released, it can be difficult for administrators to stay up to date on changes to ensure that their systems are running smoothly. But self-service infrastructure also offers a number of benefits to compensate for the troubles.

What are the Benefits of Self-Service Infrastructure?

First, self-service infrastructure provides teams with the flexibility they need to quickly develop, deploy and scale applications as needed. It eliminates the burden of non-core work from development teams so that they can work faster and more efficiently with fewer resources. For example, they don’t need to care about Terraform but have a service where they select templates that cover their needs.

This allows for more streamlined delivery processes – reducing the amount of time it takes for projects or applications from concept through production launch. Self-service infrastructure helps speed up product development cycles and reduces risk by decreasing overall time spent on debugging and testing infrastructure throughout the development stages. It removes infrastructure management from development teams entirely, even when automated, eliminating potential human errors that could lead to expensive downtime or security issues in the future.

Additionally, when combined with automation tools and DevOps best practices such as continuous integration/continuous delivery (CI/CD), it helps automate even more tedious tasks, further streamlining overall product delivery times without sacrificing quality control measures already in place.

In terms of collaboration, self-service infrastructure helps promote transparency throughout organizations by providing visibility into usage analytics while also allowing teams greater autonomy over asset utilization, which will increase productivity levels between departments.

It also helps establish better communication channels between software engineering teams who are responsible for application releases since they can deploy timely updates through automation. There’s no need for long waiting periods due to delays in approvals from vendors or other stakeholders like during traditional manual process setups.

Finally, one of the most important benefits of self-service infrastructure is scalability capabilities. Whether your business needs capacity for a few thousand users or over one hundred thousand users, it can handle any scale you require at any given moment in time so that you’re never left behind due to a lack of computing resources when demand unexpectedly increases.

Wrapping Up

Self-service infrastructure provides many opportunities for businesses to modernize their IT operations easily. With features such as scalability, automation and agile development, companies can benefit from the ability to quickly and efficiently deploy applications.

Not only does self-service IT infrastructure reduce operational costs, but it also offers greater control over resources and enables organizations to leverage their limited resources more effectively.

November 7, 2023