devopsy.kinsta.cloud

Category: DevOps Practice

  • Scaling Predictive Analytics With AIOps to Drive Next-Gen SRE

    Scaling Predictive Analytics With AIOps to Drive Next-Gen SRE

    Enterprise systems are only as valuable as they are reliable, in the sense that they don’t suffer excessive breakdowns. Otherwise, companies experience costly downtime and added stress for engineers due to the additional burden of managing issues. This critical function of ensuring systems run reliably and optimally, at production scale and with minimum human intervention, is the purview of site reliability engineering (SRE) teams.  

    The SRE professional’s job is to reduce human toil by developing and implementing reliable and highly scalable systems that optimize software and applications. Their goal is to proactively identify and, where possible, anticipate potential disruptions to minimize risk and ensure optimal system performance. However, to be successful, SRE teams require a level of enterprise-wide visibility that can be difficult to provide without the right tools to capture and analyze data at the appropriate level of detail. 

    Let’s examine how artificial intelligence for IT operations (AIOps) can support the SRE mission by connecting data enterprise-wide for unprecedented visibility and control of systems and processes, enabling auto resolution of most issues and more efficient triaging for the few remaining cases where SRE teams must collaborate around a fix.

    Challenges of Complexity and Scale in Pursuing SRE Excellence

    As more enterprises digitize their operations and move to greater automation, their IT operations must leverage all data assets skillfully in order to improve reliability and reduce human toil. The SRE profession arose to satisfy this need, with a focus on monitoring systems, accessorizing automated releases, understanding change impacts and automating some of the most common system processes. 

    To do their job effectively, SRE teams need a wide variety of data and analytics capabilities at their disposal. This includes the ability to deep dive into descriptive and diagnostic analytics to look backward and at present conditions to discover what happened in the past and why and to baseline current operations. But this is just the beginning. The true value  of SRE comes with the scaling of predictive and prescriptive analytics to draw on that historic data and apply detailed analysis to generate predictive insights into what is most likely to happen in the future; these insights are the basis for identifying proactive measures that can serve to address any potential problems and optimize those future outcomes, minimizing adverse impacts on operations 

    These analytic capabilities must be fed by robust data that comes from across the entire IT estate. Developers assigned to the SRE role face an especially strong mandate to have this holistic visibility as they plan, build, test, release, monitor and secure systems; their job is hampered to the extent that organizational silos or problems of scale get in the way of achieving that visibility. It is here that AIOps can help evolve IT operations to become more proactive and autonomous by scaling the power and reach of predictive and prescriptive analytics. 

    AIOps Empowers the SRE Mission Enterprise-Wide

    AIOps is an essential tool for the SRE community in the battle to reduce operator stress, configure IT systems to be more stable and run efficiently with less human intervention. AIOps employs artificial intelligence (AI) and machine learning (ML) for observability, context, normal behavior analysis and automated health diagnostics. This, in turn, enables anomaly detection in real-time and closed-loop automatic resolution of most issues. 

    For example, AIOps can uncover patterns showing that, 90% of the time, a particular alert in the organization’s payment system triggers a seemingly unrelated alert within 15 minutes. Moving forward, this gives SRE teams a 15-minute head start on addressing that secondary alert; the discovery forms the basis of an auto-resolution or triaging scenario that uncovers the underlying cause of both alerts and provides a permanent fix. 

    That’s just one illustration of how AIOps uses advanced analytics to discern subtle patterns in data to predict when and where problems may occur, so proactive fixes can automatically be prescribed to head off those problems. In this way, AIOps is the technology backbone that scales and automates the SRE team’s insight and control across all system assets and dependencies, a critical tool for continuous optimization of performance with minimal human intervention. 

    Conclusion: The Game-Changing Role for AIOps in SRE

    AIOps is a game-changer for the critical function of site reliability engineering. Powered by a potent blend of advanced analytics on robust data coming from systems across the enterprise, AIOps delivers unprecedented visibility and control for SRE teams in their mission to reduce toil and ensure the reliability and resiliency of enterprise systems. The result is less human intervention and enhanced value from IT systems that are made more robust, more stable and more efficient.

  • Five Great DevOps Job Opportunities

    Five Great DevOps Job Opportunities

    DevOps.com is now providing a weekly DevOps jobs report through which opportunities for DevOps professionals will be highlighted as part of an effort to better serve our audience.

    Our goal in these challenging economic times is to make it easier for DevOps professionals to advance their careers.

    Of course, the pool of available DevOps talent is still relatively constrained, so when one DevOps professional takes on a new role it tends to create an opportunity for others.

    The five job postings shared this week are selected based on the company looking to hire, the vertical industry segment and, naturally, the pay scale being offered.

    We’re also committed to providing additional insights into the state of the DevOps job market. In the meantime, for your consideration:

    LinkedIn

    Underdog Fantasy
    Remote
    Senior DevOps Engineer
    $150,000 to $240,000

    SimplyHired

    DISH Wireless
    Littleton. Colorado
    Senior DevOps Engineer
    $106,210 – $143,750

    Monster.com

    UMD Energy Solutions
    Bonita Springs, Florida
    AWS Cloud Engineer
    $100,000 to $120,000

    Indeed.com

    Medispend
    Remote
    DevOps Engineer
    $100,000 to $130,000

    Dice.com

    Venusgeo
    Clinton, New Jersey
    AWS DevOps Lead
    $120,000+

  • Building a Platform for DevOps Evolution, Part Two

    Building a Platform for DevOps Evolution, Part Two

    In the introduction to this series, we discussed the concept of change and how it applies to DevOps environments and release managers. 

    In a world of constant changes in our surroundings and user demands, DevOps has to maintain hundreds of moving pieces across large organizations of people who, by nature, are adverse to change. This requires release managers to delegate power, transition from a centralized commander to an enabler, understand different value streams and constantly experiment. 

    In this section, we’ll cover some of the ways and types of experiments DevOps organizations can conduct to improve their release management procedures and meet their objectives using DevOps principles.

    Value Stream Mapping

    Value stream mapping is a series of steps used by organizations to provide products or services their customers want or need. In the traditional sense, the release and deployment process can seem tedious, especially if teams must communicate with a release management team to schedule release slots on a calendar.

    Despite its advantages, value stream mapping has a few drawbacks. It requires a lot of people and it can be difficult to coordinate and sparsely repeated.  

    Value stream management platforms can extract the four key DevOps metrics for teams to inspect and adjust in real-time. However, there are some shortcomings with using the four metrics:

    • Lead time is characterized as time from code commit to live; however, in value stream management, we also consider the time it takes for an idea or customer request to reach value realization.
    • Deployment frequency becomes interesting when teams deploy sporadically but are still improving, but when teams attain the ability to release on demand, other metrics become more helpful.

    As a result, teams must alter metrics over time to match team ability. This is why it’s critical to empower value stream teams to discover and measure their own metrics. Another factor is that people are more likely to change when doing it for themselves rather than having it done to them.

    Lessen Batch Size

    Reduced batch size lowers risk and reduces the time an activity waits in a queue. When changes are minimal, they may be implemented quickly and with less danger of causing disruption, thereby making the distinction between the phases less significant.

    While some key metrics may be accessed through specific tools such as backlog or release automation, without a value stream management platform, it’s more difficult to gain insights into overall queueing time.

    The value stream map is automated when all aspects of the DevOps toolchain are combined, resulting in a unified system from idea through the realization that allows teams to understand their cycle time and value flow efficiency and observe how changes improve work.

    Evolving Architecture From Monolith to Microservices

    Most organizations have monolithic, tightly coupled applications because that was the best software design approach available in the past. However, as the industry evolves, release managers become increasingly conscious of the limitations of this architectural approach:

    • Large batches of work are required to build, test and deploy.
    • Dependencies must be managed, including people and systems.
    • Dependencies cause fragility.
    • Fragility causes bureaucracy (e.g., CABs).
    • Bureaucracy and handoffs hinder flow.

    Rearchitecting applications to microservices, like changing human behaviors in a system, does not happen overnight, and there are frequent bottlenecks along the way. This is often a “change or die” conversation. If the application does not become more capable of change, market disruptors will provide new experiences to consumers, and customers will jump ship for these competitors.

    A value stream management platform can reveal where restrictions, delays, and dependencies occur and how they might be reduced. As a tenet of DevOps says, “Don’t manage dependencies; break them.”

    For many companies, however, the road to systemic autonomy, or loosely coupled systems, is lengthy with many incremental steps of improvement. This method is recommended by the strangler pattern: chipping away at the monolith one bit of functionality at a time.

    During the application architecture change, we need to support value stream teams, so they don’t waste time on unexpected work caused by unknown dependencies. We also need them to see and appreciate the improvements they’ve made with the support of a value stream management platform.

    Part of the release manager’s job is to identify value stream success stories and share them with the other value streams so that local improvements can become global ones.

    Transfer the Release Management Role to the Value Stream Team

    When teams work independently on small changes, they can release them as soon as they are ready. However, moving from one state to another takes time to transition to that stage of the journey.

    In a traditional method of working, the release process is likely to be coordinated by a release manager or a team of release managers. The release manager role can be transferred to the value stream team, with the team providing access to systems that provide visibility into current releases to the release manager.

    Transparency into operations across value streams provides traditional release managers with great comfort during this transition, relieving them of the administrative strain. It allows individuals to learn new skills and refine the meaning and purpose they derive from their work.

    By taking an end-to-end and cross-value stream approach to change and improvement and ensuring blame becomes accountability, release managers can be key to the success of these improvements. Platforms for value stream management give the data needed to support the conversations that will lead to this cultural and behavioral shift.

    In this series’s third and final part, we will cover more types of experiments and DevOps best practices for achieving change and driving results. 

  • The Real Pipeline

    The Real Pipeline

    I’ve made no secret of the fact that DevOps was a game-changing advance in how the business of IT was done. But people tend to get religious about the methodology and forget the point. That’s okay, DevOps has its roots in that mentality. After years of thinking bad things about the authors of The Phoenix Project for portraying anything that slows the process–you know, like code quality or security–as bad, I have come to accept that absolutes were required to drive forward motion in the market. “We’ve always done it this way!” was king in IT, but the rest of the business was fed up. That book got the ball rolling and we’ve slowly but surely pushed back against the mentality that adding milliseconds (or even a week) to build times in the average enterprise has any impact worth mentioning. Oh, some people still claim it does, but we can convincingly make the code quality/security argument.

    It is funny to me that the first few years of DevOps were focused on “We’re gonna change your religion!”—wanting to force upheaval of the entire IT org chart without a thought to the fact that you still needed specialists in storage, ops, etc. to keep the lights on. The mentality was that development issues were a people problem. To a large extent, they were right; they just missed one people problem in the set of equations. Let’s look at the real pipeline that we all don’t talk about, but really should:

    Notice there aren’t tools in this chain; instead, there are groups of people. It is sad but true that the last two–representing acceptance testing and customer rollout–are almost always merged these days in all but a few organizations.

    This isn’t controversial; we still do the dev bit largely before the ops bit; the business has to decide what we want to offer the customer and customers are at the end of the line. And that is the key. Customers are at the end of the line. Now, I get to make you uncomfortable. Nothing in the above diagram matters except the first and last arrows. Think of it like this:

    What’s important in what we do comes down to two questions: “What does the business unit want to offer customers?” and “What do the customers expect of the business unit?” In this sense, “business unit” might be IT and “customers” might be co-workers. That changes nothing. What the business unit offers and what customers want is all that matters. The rest is implementation detail.

    Don’t believe me? It is an easy case to prove. Let’s try this one, with or without IT involvement, depending upon the problem being solved and the organization:

    The business unit determines what they want to offer, customers let their demands be known (one way or another) and the business offers the solution they think fits for the customers. As I said above, the rest is implementation details.

    What is your point?

    This leads me to the inevitable observation. Testing and security are:

    • Completely expected by users. They don’t want buggy software, and they don’t want to enter information assuming it will get nicked by hackers. There is no compromise here; users don’t “kind of want a product that works” nor do they want a product that “might be secure, but we’re not sure.”
    • Desired by the business. They have constraints on what is reasonable–the old meme of a server with the network cable cut is only funny because it shows extremes. But they absolutely do want the product or service they are offering customers to be stable and secure. “We can just spin out a new version” only works so often and for so long.

    So, your toolchains need to include stability and security tools—I would argue at every step of the process, but where and how much is really a business call. My security peeps will get upset that I called security a business call, but it is. You can choose to be non-compliant and insecure; the impact of those decisions on your organization is all on you. OTOH, of all the things that the organization (both the business generally and IT specifically) are focused on, you can afford to drop a few and get security and stability right the first time.

    Meanwhile, you’re keeping things running while we debate things like this. While I repeatedly refer to a large portion of my target audience as “implementation detail,” I’m sure you all know how much I (and many others) appreciate the rockstar job you’re doing keeping the IT org supporting the greater org.

  • Five Great DevOps Job Opportunities

    Five Great DevOps Job Opportunities

    DevOps.com is now providing a weekly DevOps jobs report through which opportunities for DevOps professionals will be highlighted as part of an effort to better serve our audience.

    Our goal in these challenging economic times is to make it easier for DevOps professionals to advance their careers.

    Of course, the pool of available DevOps talent is still relatively constrained, so when one DevOps professional takes on a new role it tends to create an opportunity for others.

    The five job postings shared this week are selected based on the company looking to hire, the vertical industry segment and, naturally, the pay scale being offered.

    We’re also committed to providing additional insights into the state of the DevOps job market. In the meantime, for your consideration:

    SimplyHired

    AmeriSave Mortgage

    Remote

    DevOps Engineer

    $99,000 to $130,000

    LinkedIn

    Sparkrock

    Remote

    Vice president of platform development

    $160,000

    Indeed.com

    Stanford University

    Redwood City, Calif.

    DevOps Engineer

    $117,000 to $148,000

    Monster.com

    Ness USA

    Remote

    DevOps Engineer

    $100,000 to $150,000

    ZipRecruiter

    LS Global Group

    Remote

    DevOps Engineer

    $115,000 to $175,000

  • Optimizing Prometheus Implementations With Managed Services

    Optimizing Prometheus Implementations With Managed Services

    Prometheus is a leading metrics-based open source monitoring solution and time series database used across the IT community to collect metrics from systems, applications and services. According to the 2021 Cloud Native Survey from the Cloud Native Computing Foundation (CNCF), the use of Prometheus in production has reached 65%, an increase of 43% year-over-year.

    “As cloud-native stacks have grown and matured, observability has become crucial for visibility into application metrics, performance, alerting, and insights,” said Chris Aniszczyk, CTO of CNCF. “Prometheus is a key element of this, enabling organizations to build a foundation of observability.”

    If you are using Prometheus, it is probably because of its many advantages including reliability, easy initial implementation and its open source nature. However, in addition to these benefits, Prometheus also brings several challenges if you are deploying and maintaining the solution on your own.

    Facing the Four Main Prometheus Challenges

    Prometheus users face four primary challenges that can be alleviated by the right service provider:

    Challenge One: Support

    Prometheus users face the dilemma posed by any open source tool: There is no vendor behind the solution to deliver technical support. It is easy to take tech support for granted, but when you are working with a technology that is not supported, even a small issue can become a substantial problem. A commercially-backed product can provide a critical level of expertise you do not have in-house, and take on responsibility for keeping the system up and running.

    Challenge Two: Maintenance

    The greatest challenge you’ll likely face with Prometheus is the amount of time your team must spend configuring and maintaining the system. 

    Just one example of how Prometheus drains your resources is the use of third-party exporters to expose metrics in the Prometheus format. These integrations are designed by members of the open source community an do not come with any support; upgrades are hit-or-miss.

    Exporters are difficult to maintain. First, there are often multiple exporter options and choosing the right one for each of your services is time-consuming. Second, you can easily have hundreds of different services running in your environment and your team can spend hours every day making sure the exporters are working properly.  

    Another maintenance issue involves PromQL, the Prometheus query language. PromQL is a custom query language, and it is very complex and difficult to learn. Having to rely on your one PromQL expert for all Prometheus queries across the organization is simply not practical or sustainable.

    With all these maintenance requirements, you will have some team members spending all their time managing and maintaining a Prometheus environment, eating away at resources that could be used to focus on business needs and innovate new applications.

    Challenge Three: Scalability

    Users tend to deploy Prometheus in silos, with a separate server for each cluster. Over time, you may have tens or even hundreds of Prometheus servers in your environment that you are not aware of. This leads to scalability issues regarding consistency, maintainability and a global view of your metrics.

    In addition, storage is a major hindrance to scaling Prometheus, because the system typically stores metrics for about two weeks. If you need to analyze metrics older than a couple of weeks, you run into a serious problem. 

    Challenge Four: Correlating Data

    Data does not have much value if you cannot correlate it and, unfortunately, Prometheus does not provide correlation for metrics to other elements in your environment like processes or cloud regions. This makes it difficult to pinpoint the source or impact of a problem. 

    Options for Solving Prometheus Challenges

    Options are available to help you with Prometheus but most do not solve the four major issues outlined above. The following are your options:

    APM: APM vendors all have some level of support for Prometheus, but this is an add-on to their platform. They offer an ingest tool that can transform Prometheus data into a format you can query natively within the APM platform. In reality, they are not helping you utilize Prometheus but instead are forcing you to work within the constraints of their system.

    Cloud tools: Some cloud hyperscalers offer solutions to support Prometheus. These tools can help with deployment and infrastructure maintenance, but you are still stuck doing all the other configuration and maintenance work. 

    Managed service: The preferred option is a Prometheus managed service, which can alleviate all four of the main challenges posed by open source Prometheus. The right optimized managed service provider delivers support, maintenance, long-term storage and data correlation.

    Must-Have Capabilities for Your Prometheus Managed Service Provider

    The following are essential capabilities you should look for when choosing a Prometheus managed service provider:

    Prometheus compatibility: You need an optimized managed service provider whose platform is designed to be natively compatible with Prometheus, consuming data in the Prometheus format and allowing you to run PromQL queries on the data.

    Simplified PromQL: To eliminate the reliance on PromQL experts for every query, a managed service provider should simplify the use of PromQL, without breaking compatibility.

    Exporter management: The right optimized managed service provider will be able to streamline the selection, deployment and management of exporters, saving you a significant amount of time and effort.

    Automated discovery: Tracking and optimizing all your services can be challenging. For this reason, a Prometheus-managed service should be able to automatically discover the services running in your environment and help guide you with configuration.

    Metric Storage: One of the shortcomings of Prometheus is the lack of long-term storage. Consequently, a Prometheus managed service provider should provide a database for long-term storage of metrics, so you can perform comparative analyses of the data, from month to month and even year-to-year.

    Prometheus compatibility: If you already use open source Prometheus, your teams are used to using things like PromQL and Grafana dashboards. You should select a managed service provider that is Prometheus-compatible so your team can use the same dashboards and queries they have always used to view the data.  

    Conclusion: Optimized Valuable Tool

    Prometheus is not a standalone solution. But it is an optimized valuable tool to collect metrics for performance analysis and it is appreciated by your architects, developers and IT Ops pros. While Prometheus may have some limitations, it is worth finding a solution to these issues; that solution is a managed Prometheus service. The right managed service provider can give you the best of both worlds—all the benefits of Prometheus without the time, resources and expense needed to run it effectively.

  • Finding Value in AI-Augmented DevOps

    Finding Value in AI-Augmented DevOps

    While artificial intelligence (AI) and machine learning (ML) are emerging technologies, we know they can help an organization parse large data sets and glean actionable insights. But do AI-infused processes actually make a difference in organizations that employ DevOps? The answer is explored in a recent survey by Tricentis. Working in partnership with Techstrong Group, Tricentis surveyed more than 2,600 DevOps practitioners across the globe about their AI-augmented DevOps processes. The results pointed to a bright future in DevOps.

    In this survey, the majority of respondents cited they’re already getting value from AI-powered DevOps processes. A third of survey respondents found AI-augmented DevOps extremely useful today and nearly half found it very useful. Although the technology is still relatively young, a 2021 report by IT infrastructure automation company Puppet Labs found that 83% of organizations are implementing DevOps processes. This rapid adoption speaks to the promise of the technology.

    Interestingly, nearly two-thirds of respondents cited that testing is the area within DevOps where AI will have the most impact. Testing is a major pain point for DevOps organizations because a scaled DevOps environment calls for complex testing scenarios and large amounts of data, and many organizations struggle to scale their automation to the level DevOps requires. Without automation, manual testing can significantly delay releases. In AI-infused DevOps processes, AI can help accelerate the authoring of tests, help testers understand where the risks are, and repair broken tests.

    AI’s Greatest Potential is in Functional Testing

    We typically think of software testing as either functional, where the program enables users to properly execute tasks, or non-functional, where the application is secure, fast and stable. Respondents were asked which disciplines of testing stood to gain most from AI and 65% of them indicated functional testing. Unit testing and UI testing were the sub-categories most often selected.

    The complexity of the functional testing process is where AI shines. Functional testing is open-ended and many permutations are required to ensure proper test coverage. AI steps in and manages those permutations, parsing large amounts of data and providing insights. Additionally, AI can help wrangle production signals that aren’t discovered in a test environment.

    Throughout the survey, respondents specifically noted AI’s promise in regard to UI testing. UI testing is perhaps the most time-intensive and manual type of testing in most organizations. With the many variations of the user experience paths that exist, AI can help mimic the actions of a real end-user, adjusting for application nuances or variations in experience. AI and ML can also boost your build process, increasing the efficiency of repetitive tasks. The ability to quickly identify and resolve issues can improve the effectiveness of your automation.

    While survey respondents see the most potential in testing, they acknowledge that AI-powered DevOps can address other business challenges. More than 40% claim that these processes can reduce the skills gap and empower junior staff to perform complex tasks, improve the customer experience, cut costs, boost innovation and increase efficiency among developers. Nearly 40% also believe that AI-powered DevOps delivers leadership the insights they need for continuous improvement.

    Automation is the Key to Scaling DevOps

    It’s worth noting that among the DevOps organizations that see benefits from AI and ML, most of them are mature organizations; they employ DevOps workflow pipelines, toolchains, automation and cloud technologies, and they’ve automated more than half of their testing. Only 21% of practitioners who are just dipping their toes into DevOps have achieved this level of automation. And even in mature organizations, testing is still the main bottleneck; only 40% of them have automated more than half of their testing.

    The key to scaling DevOps practices is developing test automation that can keep up with an increasing volume and speed of releases. But the road to DevOps isn’t always easy. Practitioners cited obstacles like a lack of tech skills (44%), insufficient budget (25%) and tool selection (19%). Organizations that want to implement DevOps need to rely on external vendors and internal resources to address these challenges and effectively scale.

    Looking to Adapt More DevOps Practices?

    So where does an organization that wants to implement DevOps start? With the big questions. It’s crucial to identify the specific business problem that you’d like to solve. What’s the gain you’re looking for? Where in the software life cycle do you expect to find a benefit? AI and ML can add business value, but they don’t solve every problem across the life cycle.

    After an organization has identified its DevOps goals, its focus should shift to automation. To reap the benefits of AI-powered testing, an organization should automate repetitive functional testing, especially unit and regression testing. Most organizations start by implementing continuous integration and continuous delivery. Once an organization has taken these steps, it will start to amass a large cache of results and artifacts generated from running the test cases, which AI can mine to increase test stability and identify recurring problems. The more you automate, the more you’ll get out of it and the faster you’ll accelerate your delivery.

    Organizations just starting out with DevOps should also understand the distinction between artificial intelligence and ML. Most people refer to ML as AI, but they’re not interchangeable, and most organizations haven’t advanced to the use of AI. A true AI is broadly powerful, while ML technologies are narrow and solve a specific domain problem that a model was trained for.

    When you’re evaluating potential vendors, there are a number of key questions to ask, including:

    • What specific AI technology is the vendor using? Is it an off-the-shelf library or something they built themselves?

    • How are they using AI? Which algorithms are in play and what approach do they take?

    • How do they train their models? Different types of patterns emerge in data sets depending on the industry, the frequency with which data is updated and the application. The data population that the vendor is training on must look like your production data, or the model could take you in the wrong direction.

    • How do they ensure that bias doesn’t come into play? If they train models in a different domain or vertical, for example, you could be an outlier for that vendor.

    • How do they handle failure? How do they monitor for miscategorization and misapplication of the AI in a way that lets you trust the data?

    • Will your data be used to train and improve your learning models? If so, does that raise any privacy concerns? Expectations around data privacy should be explicit and leave no questions.

    • Are there other customers in your vertical with a similar business profile that are getting the benefits you’re looking for? Can the vendor provide examples?

    What Does AI-Infused DevOps Look Like in Practice?

    Consider a financial services organization that has more than 20 applications to manage. With so many applications, it’s necessary to conduct significant UI testing and the tests must be stable. One of the biggest challenges to UI testing lies in locating various web elements; most people start with manual script-based automation, but it can be challenging and time-consuming to create and maintain. Employing AI and smart locators to identify these elements can save time and drastically cut down on UI testing maintenance.

    Survey respondents were asked how they’re currently using AI to augment their testing processes. Thirty-seven percent say they’re accelerating the creation of their automated test cases, while 44% focus their testing on the areas of highest risk. Almost half are able to reduce test case maintenance with self-healing and 43% use AI to identify the root cause of failed tests. Thirty-one percent claim that AI offers insights into test process improvements and 34% use AI to help identify tests to run based on changes within the application.

    Where is AI-Powered DevOps Going Next?

    Going forward, AI will become more and more embedded in testing processes. Great progress with off-the-shelf vendors is being made, and their products are a lot more accessible than they used to be. As the tooling continues to mature, these products will reduce the need for specialized skills. Internal initiatives like data lakes or data warehouses will persist, but there will be less need to build large data science teams to get the advantages of AI.

  • Heat Cooks Twitter DC | AI Will Kill All Humans | Patreon Layoffs, CSAM Claim

    Heat Cooks Twitter DC | AI Will Kill All Humans | Patreon Layoffs, CSAM Claim

    Welcome to The Long View—where we peruse the news of the week and strip it to the essentials. Let’s work out what really matters.

    This week: Twitter is in a “non-redundant state” thanks to a hot summer, AI is likely to eliminate us, and Patreon fires 80 staff amid nasty allegations (TW: Child safety). (more…)

  • Linus Torvalds Signals Support for Memory-Safe Rust Language

    Linus Torvalds Signals Support for Memory-Safe Rust Language

    Linus Torvalds told attendees of the Open Source Summit Europe conference today that he is hopeful the foundational work needed to add support for the Rust programming language will be laid in a forthcoming version 6.1 of the Linux operating system.

    It would take several additional releases of Linux to add full support for Rust, but the ability to more easily build and deploy applications written in Rust will go a long way toward improving the overall state of application security on Linux platforms.

    The Open Source Security Foundation (OpenSSF), an arm of the Linux Foundation, has called for replacing programming languages like Java that are not memory-safe as part of a larger plan to secure open source software supply chains. Rust is a memory-safe language that employs a compiler to track the ownership of values that can be used once and a borrow checker that manages how data is used without relying on traditional garbage collection techniques.

    Earlier this week, the Rust Foundation allied with OpenSSF and JFrog to address other security issues that might arise as applications are developed in Rust.

    It’s not clear just how many developers are adopting Rust, but Torvalds said he personally was looking forward to learning the Rust syntax—using a recently-acquired laptop based on an Arm processor—should the maintainers of Linux approve the effort. However, Torvalds also left open the possibility that adding support for Rust might not be as successful as initially hoped.

    Torvalds also bemoaned the overall state of cybersecurity transparency at the operating system level. Many of the bugs that are encountered at the hardware level cannot be shared with the entire open source community, so any fix to those efforts is not subject to the same level of peer review as the rest of the operating system. The overall situation is improving, but it’s still very painful from a development perspective, he said. Secrecy is counter to the open source ethos, he said.

    As new applications are built using a memory-safe programming language, the overall security posture of an organization will steadily improve. The challenge, of course, is that the number of developers that know how to build applications using those languages is still relatively small. Of course, replacing trillions of lines of code that have already been constructed using a variety of legacy non-memory-safe languages represents a gargantuan task that might never be fully completed—but shifting to a modern programming language is a major step in the right direction when it comes to application security.

    There may even come a day when the way applications are coded is a lot more secure by default than it is today. That may not necessarily resolve every application security issue, but as the programming languages become inherently more secure, the pressure on developers to become cybersecurity experts should ease—assuming, of course, they are willing to learn a new way to write code.

  • Where Should I Store My IaC?

    Where Should I Store My IaC?

    The most successful software companies rely on repeatability, auditability and simplicity when building solutions. The emergence of infrastructure-as-code (IaC) has empowered developers to apply these practices to infrastructure allocation. What are the best practices for storing IaC? Should IaC code be stored with the application and feature code? Should it be stored separately in its own application-specific repo? Or should all IaC code for all applications in an organization reside in the same repo? 

    Let’s explore these questions, determine the tradeoffs between each solution and create guidelines that consider your organization’s needs.

    The Problem With Storing IaC With Feature Code

    The naive option for storing IaC is to keep all IaC code within the application repository that consumes this IaC code as part of its CI/CD pipeline. It’s easy to set up, doesn’t require complex permission allocation and will be fully visible to developers who work on this codebase, thus making it easy to retrieve context. This ease is why 20% of polled developers state they always store feature code and IaC in the same repo. However, there are a number of problems with holding IaC code with feature code.

    Software development best practices include minimizing redundancy. Storing all IaC code within the dependent application’s codebase violates best practices by requiring the infrastructure-as-code to contain not just application-specific allocation information but also all foundational infrastructure. Organizations may classify common infrastructure shared across multiple applications and ecosystems as foundational infrastructure. Examples of such base, foundational infrastructure include Kubernetes clusters and log storage accounts. 

    In addition, storing IaC code with feature code introduces risk. Allowing developers admin-level access to all deployment scripts needed to bring code to production violates the principle of least privilege and makes lateral movement easier for attackers who manage to hijack a developer account.    

    Organizations benefit from a degree of secrecy regarding their deployment processes; the deployment pipeline and its configurations are bespoke and attackers who have a mapping of the services an organization uses can more easily architect supply chain attacks. Storing IaC with feature code makes it so that a code leak reveals all CI/CD configurations for not just the application, but the whole organization. The last thing an organization should do is perform an attacker’s reconnaissance for them, but that’s exactly what happens in the above situation.

    Storing infrastructure-as-code alongside product code may be easier to set up initially, but violates the principle of least privilege and zero-trust principles. This method of storing code introduces unnecessary risk–there must be a better solution.

    Storing All IaC Code Together is Not the Answer

    Since storing IaC with feature code in a common repo isn’t the optimal solution, what about storing all IaC in a dedicated IaC repo? This would allow DevOps and security teams to implement granular access controls and make code redundancy easily identifiable and preventable. However, this organizational structure introduces its own problems.

    When all IaC is stored in a separate, shared repository, it becomes difficult to match feature changes to IaC changes. This scenario introduces friction to the development processes, puts a damper on developer velocity and increases tension between development teams, DevOps teams and security teams. 

    Storing all IaC in a common repository gives developers virtually no control over security decisions affecting their workflows. Between the lack of ownership and the additional steps needed to modify any code-specific deployment scripts, storing all IaC together prevents developers from quickly changing IaC and it unnecessarily hurts the organization.

    IaC Falls Into Two Categories: Base and Feature-Specific

    IaC and other CI/CD scripts ultimately fall into two categories: base and feature. 

    Base deployment scripts may be described as code used broadly by an organization for, among other things, routine deployment actions or to interface with shared resources. Separating this code empowers organizations to allocate base IaC ownership and maintenance responsibilities to their DevOps team, which is well suited to maintain shared infrastructure.

    Feature IaC scripts are the bits of code used to provision and manage resources associated with specific software. Developers frequently interact with the deployment scripts tied to their deliverables; changing release steps, modifying gateways, updating server configurations and many other needs necessitate different handling protocols for feature IaC versus base IaC.

    Now that we understand the difference between the feature and base IaC, we can define proper management strategies for each.

    The Solution: A Hybrid Approach for Storing IaC

    Best practices call for categorizing feature IaC and base IaC differently and allocating the responsibilities to separate teams to maximize security while not adding unnecessary friction to the development and release process. Base IaC should be managed by DevOps teams, while feature-specific IaC should be handled by the code owners of the managed software.

    For your organization, this starts with inventorying the organization’s systems and services. Gaining visibility into how these resources are shared helps inform business decisions regarding IaC code storage, helping minimize redundancies within deployment protocols. Visibility is integral to defining which IaC is foundational to software deployed by the organization versus IaC which is specific to certain applications, libraries and other software deliverables.

    Helping DevSecOps Teams Manage IaC and More

    Best practices don’t mean much if organizations can’t enforce them. Some concluding tips: 

    • Track resources, including registries, repositories, builds and cloud/IaC deployments
    • Implement granular access controls, enabling the appropriate allocations of ownership depending on whether the resource is meant to be shared or application-specific
    • Provide visibility into the use of resources which empowers organizations to adhere to the principle of least privilege without adding unnecessary friction to the development process
    • Provide a comprehensive asset inventory by integrating directly with the components of your organization’s software supply chain

    These integrations allow complex insights to be derived that are simply not possible with other security tools. In addition, these integrations allow for constant monitoring for security updates, thus reducing the time between a zero-day being discovered and patching the vulnerability.