devopsy.kinsta.cloud

Category: Enterprise DevOps

  • Low-Hanging Fruit, 2023 Edition: Part One

    Low-Hanging Fruit, 2023 Edition: Part One

    We’ve been cranking it out, haven’t we? Generating cool new toys for the business, creating new infrastructure to host those new toys and trying our best to secure all this new stuff while maintaining both our sanity and our cadence. And we’re doing it. Kind of.

    One thing we’re naturally lagging behind on is security. While we don’t purposely shortchange security in our DevOps processes these days, it always takes a bit to figure out the best way to secure new technologies after they’re implemented. And we’re implementing a ton of new technologies. So our security staff (and increasingly, DevOps/SRE teams) is rushing to play catch-up. One area where this is screamingly apparent is APIs. We’re spewing them across the organization, often without cataloging or responsibility. This makes sense, as in many cases, the API is just an evolution of the function call. “I need a service, it is a good generic service, let me throw it up and we can use it from wherever the client ends up deployed,” type of thing. The problem is that leaves a ton of attack surface not truly protected. Because without cataloging or responsibility, there is no way for security to know about many of these APIs.

    API Scanning Technology

    That brings us to today’s topic. If you haven’t done so yet, it is past time to look at API scanning technology. It comes standalone and bundled with pretty much any API security product on the market and is increasingly bundled with other development security or even development tools. And the whole point is to get a list of APIs. Oh, most commercial implementations do something with that list–include it in static testing, drive it into WAFs, whatever—but just having the ability to inventory what is running on your network is reason enough to stop doing less important stuff and figure out what your exposure is. And the tools make it easy. They come in two flavors, and we should consider both.

    Static API Scanning

    The first type digs through source code and API specs to come up with the list of what is supposed to be running on the network. The big weakness of this method is that it only knows what it is told. This is static API scanning.

    Dynamic Scanning

    The second watches network traffic for API calls and then documents them, possibly even generating API specs for you. The big weakness of this method is that it only discovers what is being accessed on the network while it is watching traffic. Currently, unaccessed APIs are not caught, nor are ones on unreachable networks. Unsurprisingly, this is dynamic scanning.

    As usual, the two together are more powerful than apart. By implementing both technologies, an organization can work toward a hardcore comparison of what should be running on the network versus what is running on the network. This goes beyond detecting unreported APIs, it also means that zombie APIs or runtime drift can be discovered by comparing what is there to what should be there.

    If you’re not doing this, consider adding it to the toolbox. Setup is not onerous, and the reports are lists of APIs and parameters that can be reviewed at leisure. Some commercial products that offer these two functionalities take it quite a way beyond list generation, and if comparisons are made, work hard to understand just how much each toolset can do.

    And keep rocking it. We need more tools to lock down the new perimeter precisely because you and your team are tackling new technologies and platforms to better serve the business. We are increasingly seeing security tooling catch up with the broad selection now available, so go implement them.

  • 4 Salesforce DevOps Adoption Challenges

    4 Salesforce DevOps Adoption Challenges

    In today’s ultra-competitive digital landscape, adopting DevOps practices has become crucial for organizations looking to accelerate their development processes and enhance software delivery. Salesforce is no exception to this trend.

    Salesforce DevOps essentially refers to applying DevOps principles and practices to the development and deployment of applications and customizations on the Salesforce platform. When done right, it integrates development and operations teams and offers numerous benefits, including better collaboration, faster release cycles and improved code quality.

    Furthermore, Salesforce DevOps empowers organizations to adapt rapidly to changing business requirements, respond to complex customer needs faster and improve their experience via process automation (for instance, streamlining the product configuration and price quote generation process with CPQ automation) and drive innovation more efficiently on the Salesforce platform.

    However, while the advantages of Salesforce DevOps are evident, its adoption comes with its fair share of challenges. In this article, we will dive into the four core hurdles that organizations face when embracing Salesforce DevOps. Moreover, we will provide actionable tips to tackle these challenges head-on, helping you ensure a successful Salesforce DevOps adoption journey.

    1. Siloed Development and Operations Teams

    DevOps isn’t just about technology and automation; it is also about people. In the realm of Salesforce DevOps, one of the most significant challenges that organizations encounter is the existence of siloed development and operations teams. Traditionally, these two teams have operated independently, often with minimal interaction, leading to fragmented workflows and hindered collaboration.

    This disconnect between dev and ops can impede the delivery of high-quality Salesforce applications and features. The lack of timely communication and coordination can result in misaligned priorities, delays in deployment, and an increased risk of errors during the release process.

    Moreover, siloed teams can stifle innovation and limit the potential for continuous improvement. The development team may create features that do not align with the operational requirements or may not be feasible to deploy efficiently. On the other hand, the operations team might struggle to understand the intricacies of the development process, hindering their ability to provide valuable insights or feedback.

    To overcome the challenge of siloed development and operations teams in Salesforce DevOps, organizations must foster a culture of collaboration and shared responsibility. Here’s how:

    • Promote cross-functional collaboration: Encourage regular meetings and knowledge-sharing sessions between development and operations teams. Foster an environment where team members feel comfortable discussing challenges, sharing insights, and jointly working towards solutions.
    • Break down communication barriers: Utilize collaboration tools such as Slack and Asana that facilitate seamless communication and information sharing between teams. This can help reduce misunderstandings and keep everyone informed.
    • Align incentives and goals: Create shared objectives and metrics that encourage both development and operations teams to work together towards common goals such as improving product-market fit or feature adoption. Recognize and reward collaborative efforts to reinforce the importance of cross-functional teamwork.

    2. Version Control and Source Code Management

    Effective version control and source code management are vital pillars of successful Salesforce DevOps practices. However, many organizations with layered dev teams find it tricky, leading to complexities, inefficiencies, and potential risks in their development processes.

    Version control is essential for tracking changes to Salesforce customizations and configurations over time. Without proper versioning, development teams may find it difficult to collaborate, understand changes made by others, or revert to previous states in case of issues. Additionally, source code management allows for better organization and control of codebases, ensuring consistent and reliable deployments.

    To tackle the challenges related to version control and source code management in Salesforce DevOps, teams should implement best practices and use appropriate tools. Here’s how:

    • Adopt a version control system: Implement a robust version control system, such as Git, specifically tailored for Salesforce projects. Git allows teams to track changes, manage branches, and collaborate efficiently.
    • Establish clear branching strategies: Define clear and consistent branching strategies for different development scenarios, such as feature development, bug fixes, and releases. This helps manage parallel development efforts and ensures smooth collaboration.
    • Encourage code reviews and collaboration: Conduct regular code reviews to ensure code quality, identify potential issues, and ensure compliance with best practices. Encouraging collaboration during code reviews fosters knowledge sharing and enhances the overall development process.
    • Automate deployments and rollbacks: Integrate version control with your Continuous Integration and Continuous Deployment (CI/CD) pipeline to automate deployments. This reduces the risk of manual errors and facilitates easy rollbacks in case of issues.
    • Educate and train development teams: Provide training on version control best practices and tools to ensure all team members are familiar with the processes and benefits of using version control effectively.

    3. Continuous Integration and Deployment

    Continuous Integration and Deployment (CI/CD) are core principles in the DevOps methodology that aim to automate and streamline the software delivery process. However, implementing effective CI/CD practices for Salesforce development can be a daunting task for large-scale enterprise companies, presenting them with various challenges along the way.

    One of the primary challenges in CI/CD for Salesforce is the complexity of customizations and configurations. Salesforce projects often involve multiple components, such as Apex classes, triggers, Visualforce pages, and metadata configurations. Coordinating the integration and deployment of these components in a seamless and automated manner can be intricate, leading to potential bottlenecks and delays.

    Moreover, organizations may encounter difficulties in ensuring the stability and reliability of their CI/CD pipelines. Incomplete or inaccurate tests, improper versioning, and inadequate rollback mechanisms can result in the introduction of bugs or errors into the production environment.

    Here are a few ways to combat this challenge:

    • Automate the build and deployment process: Use Salesforce-specific CI/CD tools that automate the build and deployment process, reducing manual intervention and ensuring consistency in deployments.
    • Implement comprehensive testing: Incorporate automated testing into the CI/CD pipeline to validate changes and ensure code quality. This includes unit tests, integration tests and regression tests to detect issues early in the development life cycle.
    • Use sandboxes for testing and staging: Utilize Salesforce sandboxes to test changes in environments that mimic the production setup. This allows teams to identify and address issues before deploying to the production org.
    • Establish version control for metadata: Use version control for Salesforce metadata to keep track of changes and enable seamless collaboration among developers working on different features.
    • Monitor and optimize pipeline performance: Continuously monitor the CI/CD pipeline to identify areas of improvement and optimize the deployment process for speed and efficiency.

    4. Test Automation and Code Quality

    Test automation and code quality are critical aspects of Salesforce DevOps that directly impact the reliability and performance of applications. However, organizations often have a tough time when it comes to implementing comprehensive test automation and ensuring consistently high code quality in their Salesforce projects.

    Manual testing can be time-consuming, error-prone and insufficient to keep up with the speed of Agile and DevOps development practices. The lack of automated testing can lead to delays in identifying bugs, lower confidence in code changes and a higher risk of regression issues when deploying new features.

    Furthermore, maintaining code quality becomes challenging as Salesforce projects grow in size and complexity. Without adequate quality control measures, development teams may introduce code that does not adhere to best practices, leading to suboptimal performance, security vulnerabilities and increased technical debt.

    Here are a few tips to tackle this challenge:

    • Implement automated testing frameworks: Utilize Salesforce’s built-in testing capabilities, such as Apex unit tests and Lightning Web Component (LWC) tests, to automate the validation of code changes and functionalities.
    • Adopt test automation tools: Consider leveraging third-party test automation tools that can extend and enhance your testing capabilities, such as Selenium for end-to-end testing of Salesforce applications.
    • Create a comprehensive test suite: Develop a comprehensive suite of test cases that cover different aspects of your Salesforce application, including positive and negative scenarios, boundary testing, and integration testing.
    • Integrate testing into CI/CD pipelines: Incorporate automated testing into your CI/CD pipeline to ensure that code changes are automatically tested before being deployed to production.
    • Conduct regular code reviews: As mentioned earlier, establish a culture of code reviews to assess code quality, identify potential issues and ensure adherence to coding standards and best practices.
    • Invest in code analysis tools: Use code analysis tools such as SonarQube and CodeScan to identify potential bugs, security vulnerabilities and maintainability issues in your Salesforce codebase.

    Wrapping up

    Embracing Salesforce DevOps offers numerous benefits for organizations, including increased collaboration, faster delivery cycles, and improved software quality.

    From breaking down silos between development and operations teams to establishing robust version control and source code management practices, addressing CI/CD complexities, implementing test automation, and prioritizing code quality, a lot goes into driving successful Salesforce DevOps adoption. Armed with the solutions, it’s time to implement them, adapt them to your organization’s unique needs, and embark on a journey of unmatched quality and collaboration in your Salesforce projects.

  • How to Prevent Zombie APIs

    How to Prevent Zombie APIs

    There is a growing reliance on web APIs. A full 92% of organizations plan to increase their investments into APIs over the next year, the 2023 Postman State of the API Report found. APIs are powering everything from internal microservices strategies to partner strategies and fully-fledged products.

    Yet, this newfound API sprawl has consequences; a looming threat that might rise from the grave to haunt you …

    I’m talking, of course, about zombie APIs. Zombie APIs are endpoints that are no longer maintained yet are still active. They may be unused endpoints, old features never officially deprecated or forgotten development or testing environments. Regardless, zombie APIs are a type of technical debt that could pose a legitimate threat if left to rot.

    The sudden growth of APIs, along with immature inventory management and a lack of internal communication, can contribute to creating countless shadowy endpoints. Below, I’ll review some of the issues that zombie APIs might cause and consider some actionable steps DevOps teams can take to prevent them.

    Understanding Zombie API Risks

    Forgotten APIs can cause a number of issues. For example, suppose an insecure testing environment was accidentally left exposed to the public. An attacker using automated endpoint scanning could find authentication and authorization holes to exploit, potentially exfiltrating sensitive data.

    It’s also a bad idea to let any piece of technology go unmaintained, given the daily influx of new exploits and vulnerabilities. This point is accentuated by the connected nature of APIs, which may plug into a vast network of databases and allow both read and write privileges to the backend infrastructure.

    Insecure APIs can cause a business both economic harm and damage reputation. Just consider the high number of API-related data breaches in recent years, which occurred at companies like Optus, Dropbox, Twitter and Zendesk. It’s even estimated that somewhere between $12 billion to $23 billion was lost in 2022 due to compromises in web APIs.

    Zombie APIs are an easy target for these sorts of cyberattacks because they tend to have insecure designs that are easier to crack. They also tend to use immature monitoring systems, which means hackers can act under the radar more easily than with actively used endpoints.

    Tips to Avoid Zombie APIs

    Keep an Active Inventory of APIs

    As they say, you can’t secure what you don’t know about. In fact, OWASP lists improper inventory management as one of the ten top API security risks. Therefore, proper inventory management is a good first step to avoiding zombie APIs. Scan your service portfolio for forgotten endpoints, catalog them and update the catalog regularly.

    It’s also a good practice to thoroughly document your services to aid in internal discoverability. If possible, adopting specification-first practices can also ensure the documentation and implementation are based on a single source of truth, thus avoiding mismatched versions and documentation drift.

    “The key to preventing zombie APIs is visibility,” said Jeremy Snyder, CEO, FireTail. “Many organizations lack ongoing, up-to-date inventories of all the APIs that are running in either production or staging environments.”

    Use Proper Versioning and Life Cycle Strategies

    Another method to reduce the chance of zombie APIs is to adopt mature API life cycle management practices. The chosen API versioning strategy should be clearly defined and codified. And versioning changes should be clearly communicated to all stakeholders.

    On that note, it’s just as important to plan out a sunsetting, retirement and deprecation strategy from the onset. Then, remove deprecated services when they have reached their end of life! This is not a glamorous task, but it should be baked into someone’s assignments.

    “Good cyber hygiene is an often overlooked strategy,” explained Snyder. “A second requirement is to have a policy around versioning and how many versions of an API can exist at any given moment. For instance, that policy may state that only the latest version can exist or the latest plus the previous. However, this relies on having visibility in order to maintain compliance.”

    Other areas of proper life cycle management include employing governance around development and deployment and utilizing automated testing in continuous deployment (CI/CD) pipelines. Both of these could help set policies and security requirements, which should also bolster API safety.

    Share Knowledge of Internal and Third-Party Services

    Tribal knowledge is the antithesis of DevOps culture. As such, it’s a good practice to de-silo and share internal knowledge of API-powered services to limit the propagation of rogue endpoints. This could be done through private network repositories or even an internal Postman Collection or Workspace.

    “Internal processes like organizational coordination and communication are critical,” added Snyder. “If teams don’t tell each other about new APIs being developed and launched into production, security teams can’t assess or audit them. Conversely, if app teams don’t know about network infrastructure resources designed to cover various API security aspects, developers don’t know what security controls need to be baked into the API code.”

    Visibility is Required to Avoid Zombie APIs

    A recent CISO survey revealed that API security was the number one concern for CISOs. For many reasons, web APIs are still low-hanging fruit for attackers. This is underscored by the 400% spike in malicious API attack traffic recorded by Salt Labs earlier this year.

    Many of these attacks, we’ve also learned, are occurring over authenticated APIs. This shows that greater accountability and visibility are required to begin to stem the tide of API-related security incidents.

    To reach an adequate level of visibility, leaders should consider adopting tooling that helps determine if there are outdated APIs or APIs with multiple live versions, said Snyder. “These tools should help display what security live APIs have and provide recommendations around how to fix them, if not providing the necessary code or modules to secure the APIs directly.”

  • 2024—Year of the Linux Desktop? ChromeOS Reflects its Inner Penguin ¦ GNOME Rethink

    2024—Year of the Linux Desktop? ChromeOS Reflects its Inner Penguin ¦ GNOME Rethink

    Welcome to The Long View—where we peruse the news of the week and strip it to the essentials. Let’s work out what really matters.

    This week: Can the Linux desktop installed base ever break the mythical 10% barrier? Google has been refactoring ChromeOS, and GNOME is working on new window manager ideas. (more…)

  • Using AI to Shift Production Wisdom Left

    Using AI to Shift Production Wisdom Left

    In my recent blog, Revolutionizing the Nine Pillars of SRE with AI-Engineered Tools, I indicated that AI could assist in shifting the wisdom of production to the left. In this blog, I’ll explain in more detail how AI-engineered tools can be used to improve the pillar that I call SRE culture.

    AI Use Cases for Shifting SRE Culture Left

    Shifting the “wisdom of production” to the left is all about bringing production considerations into the development process as early as possible. Here are some use cases where AI can help facilitate this shift, along with specific challenges and AI tools that can assist.

    Predicting System Behavior: Predicting system behavior under various conditions can be difficult, especially as systems grow more complex. AI can help by analyzing historical data and predicting system behavior. Tools like Splunk can use machine learning to provide these predictions and facilitate earlier decision-making.

    Automated Code Reviews: Manually reviewing code for potential production issues can be time-consuming and error-prone. AI can automate this process, identifying potential issues more quickly and accurately. Tools like DeepCode or Codota use AI to analyze code and identify potential issues.

    Security Vulnerability Detection: Identifying security vulnerabilities early in the development process can prevent costly and damaging breaches in production. AI can analyze code to identify potential vulnerabilities that might be missed by human reviewers. Tools like Snyk or WhiteSource can help with this.

    Performance Optimization: Predicting how code changes will impact performance can be challenging. AI can analyze changes and predict their impact on system performance. Tools like Akamas leverage AI to automate the performance optimization process.

    Dependency Analysis: Identifying how changes in one part of a system will impact other parts can be difficult. AI can analyze system dependencies and predict the impact of changes. Tools like FOSSA can help in automating dependency analysis and identifying potential issues.

    Resource Allocation: Predicting resource needs for new features or changes can help prevent resource shortages in production. AI can analyze resource usage patterns and predict future needs. Tools like Turbonomic or Densify use AI to optimize resource allocation.

    User Experience Optimization: Predicting how changes will impact the user experience can help prevent negative impacts in production. AI can analyze user behavior and predict the impact of changes on user experience. Tools like FullStory or Crazy Egg can provide AI-driven insights into user behavior.

    RoadMap to SRE Wisdom

    A practical implementation roadmap for introducing AI tools to help shift wisdom to the left may include the following steps.

    Identify Needs and Goals: Start by identifying the areas where shifting wisdom to the left could have the most impact. This might include areas where issues are frequently discovered late in the development process or where manual processes are time-consuming or error-prone. Define clear goals for what you hope to achieve by introducing AI in these areas.

    Research Tools: Once you’ve identified your needs and goals, research the AI tools available that could help meet those needs. Consider factors like the tool’s capabilities, integration with your existing systems, cost and support options.

    Evaluate Tools: Based on your research, select a few tools to evaluate more thoroughly. This could involve setting up trials or demos, speaking with the tool’s developers or other users and assessing how well the tool meets your needs.

    Pilot Implementation: Once you’ve selected a tool, start with a pilot implementation on a small scale. This will allow you to learn how to use the tool, identify any potential issues or obstacles and understand the tool’s impact before rolling it out more broadly.

    Training: Provide training for your team on the new tool. This could include formal training sessions as well as ongoing support as they start using the tool.

    Monitor and Adjust: After the tool has been implemented, monitor its use and impact. This could involve tracking metrics related to your initial goals, as well as soliciting feedback from your team. Use this information to make any necessary adjustments to your use of the tool.

    Expand Use: Once you’re satisfied with the results of your pilot implementation, expand the use of the tool to other areas of your organization. Continue to monitor its use and impact and make any necessary adjustments.

    Repeat: Technology is constantly evolving, and there may be new tools or features that could further improve your ability to shift wisdom to the left. Periodically revisit your needs and goals, and repeat this process to continuously improve your processes.

    Summary

    Shifting the wisdom of production to the left is an essential strategy for modern IT organizations, but it’s a journey that requires thoughtful planning, the right tools, and a commitment to continuous learning and improvement. As we’ve explored, there are several compelling use cases for leveraging AI in this transformative process. Whether it’s predicting system behavior, automating code reviews, detecting security vulnerabilities, optimizing performance, analyzing dependencies, managing resources or enhancing the user experience, AI tools are helping organizations tackle these challenges more efficiently and effectively.

    While embarking on this journey may seem daunting, a well-crafted implementation roadmap can help make the process manageable and rewarding. By identifying specific needs and goals, researching and evaluating the right tools, piloting their implementation, providing comprehensive training and consistently monitoring and adjusting their usage, organizations can successfully adopt AI tools to shift the wisdom of production to the left. With continued effort and commitment, this transformation can lead to more informed decision-making, more efficient processes and, ultimately, better outcomes in production.

  • A Distributed Workforce is the New Normal

    A Distributed Workforce is the New Normal

    The pandemic emphasized and improved what we have known for decades: A team does not need to be centrally located to be productive. This varies based on the people involved and roles being fulfilled, but honestly, lots of people can work just fine from home, a coffee shop, the pool …

    And while we could debate the wisdom and usefulness of lockdowns forever, they did drive organizations that had never considered distributed work to actually take the plunge. Many are returning to centralized organizations, but a significant number of employees have moved to where they would like to live to work from home, and some of those employees will not be back.

    I normally avoid reference to any contentious societal discussion, but the lockdowns and pushback directly impact us in IT. The circle that was our network had already expanded to include public cloud, hosting providers and SaaS providers. Now it includes all of those plus some–perhaps only one or two, perhaps many–home offices. And that is unlikely to change, no matter how hard an organization tries. If nothing else, motivated individuals will want to work from home after hours, and if they are salaried, the organization will support that approach—meaning we in IT will have to support that approach.

    A deep look at what that implies is necessary. One of the things that having a sudden “work from home or shut down” mandate imposed upon companies did was make security an afterthought. For the first couple of months of 100%-remote work, we saw crazy numbers of access misconfigurations that caused breaches. That’s just the easy one. Your source is in a publicly accessible repository. Are the tools to access that source secure? Are you sure? What about SaaS? Is there a method other than user-generated passwords required to access each SaaS? Do you even know what SaaS systems business users have adopted? Does the organization use password management? It is necessary since SSO isn’t covering everything, and really can’t.

    The same is true for accessibility. Management teams started with, “Let’s get people working,” because the alternative to people working was layoffs or even corporate bankruptcy. If you paid any attention at all in that timeframe, you heard the horror stories of IT going to employees’ houses and setting up or troubleshooting corporate desktops that went home. It was an ugly time, and only afterward did organizations start to consider options for handling communications. Is it policy to have someone go to a nearby coffee shop when their connection is down, or is that a day off? Can they work without connectivity and then upload afterward? Most organizations still don’t have coherent policies in this regard, allowing teams to decide. In fairness, not every employee is great at working from home, so it is possibly the best option to have team leads deal with failed communications issues … But a corporate policy or guidelines should still be developed, and technical solutions like satellite backup should also be considered.

    Organizations rushed to WFH and then rushed to return-to-office (RTO). Your job is to make sure that, with all of the rushing, things are stable and secure. You’ve been rocking it all along—don’t let shifting corporate priorities drag you down now.

  • Overture Maps’ Challenge to Google ¦ Frontier Model Forum’s AI Safety Shtick

    Overture Maps’ Challenge to Google ¦ Frontier Model Forum’s AI Safety Shtick

    Welcome to The Long View—where we peruse the news of the week and strip it to the essentials. Let’s work out what really matters.

    This week: A big-tech consortium goes toe-to-toe with Google Maps, and a big-tech consortium tries to fool AI regulators. (more…)

  • Upgrade SRE Performance Management With AI

    Upgrade SRE Performance Management With AI

    In my recent blog, Revolutionizing the Nine Pillars of SRE with AI-Engineered Tools, I indicated that AI could analyze application and infrastructure performance data to identify optimization opportunities and predict capacity requirements. In this blog, I explain in more detail how AI-engineered tools can be used to improve the pillar that I call performance management of apps and infrastructure.

    AI Use Cases for Performance Management

    Performance management in the SRE context generally refers to monitoring, analyzing and optimizing the performance of IT systems to ensure they are functioning efficiently and effectively to meet the needs of users and the business.

    Performance Monitoring and Analysis: With the complexity and scale of modern IT systems, manually sifting through performance data to find issues can be like finding a needle in a haystack. AI can help identify patterns and anomalies that might indicate a performance problem. Tools like Dynatrace and Datadog use AI to analyze performance data and automatically alert you to potential issues.

    Capacity Planning: Predicting future capacity needs is often more of an art than a science, but AI can help make these predictions more accurate. Machine learning algorithms can analyze historical usage data and predict future capacity needs based on patterns and trends in this data. A tool like Amazon Forecast can generate accurate capacity forecasts based on time-series data.

    Load Testing and Stress Testing: AI can enhance these testing techniques by dynamically generating realistic load and stress scenarios based on real-world usage patterns. AI can also analyze the results of these tests to identify potential bottlenecks. Tools like BlazeMeter with Taurus provide capabilities for AI-enhanced load and stress testing.

    Performance Optimization: AI can identify optimization opportunities that humans might miss. Machine learning algorithms can analyze performance data to identify inefficient operations or configurations that are impacting performance. Tools like Akamas leverage AI to automate the performance optimization process.

    Performance Modeling: Traditional performance modeling techniques can struggle to accurately model the behavior of complex, distributed systems. AI can analyze historical performance data to create more accurate models of system behavior. Tools like BMC’s TrueSight Capacity Optimization employ AI to enhance performance modeling capabilities.

    Overcoming Challenges

    Implementing AI in any area brings its own set of challenges, and performance management is no exception. Here are some potential challenges and ways to overcome them:

    Data Quality and Management: AI relies heavily on data to provide accurate predictions and optimizations. If the data is inaccurate, incomplete or not representative of the system’s behavior, the AI’s effectiveness will be limited. To overcome this, it’s important to implement robust data management and governance practices. Regularly review and clean your data to ensure its accurate and relevant.

    Skills Gap: Implementing AI often requires a specific set of skills in data science, machine learning and AI, which may not be present in existing SRE teams. Investing in training and upskilling for existing team members, as well as considering hiring or partnering with AI experts, can help fill this gap.

    Resistance to Change: As with any major change, there can be resistance from team members who are comfortable with existing practices. Open communication about the benefits of AI and how it can make their jobs easier and providing the necessary training can help overcome this resistance.

    Integration with Existing Systems: AI tools need to be able to integrate with existing systems to access necessary data and take actions based on its analysis. Choosing AI tools with robust integration capabilities and planning for potential integration challenges can help overcome this issue.

    Cost: Implementing AI can require significant investment in tools, infrastructure and skills. Careful planning and budgeting as well as calculating potential ROI from the improved performance management can help justify these costs.

    Security and Privacy: AI tools need access to a lot of data, which can raise security and privacy concerns. It’s important to choose AI tools that follow best practices for data security and privacy and to conduct regular security audits.

    Summary

    Performance management is a critical part of site reliability engineering, ensuring that systems function at their optimal efficiency and efficacy. With the integration of AI, this facet of SRE is getting an unprecedented boost. From using AI in performance monitoring to identify patterns and anomalies to employing machine learning for accurate capacity predictions, AI is redefining how SRE practices are carried out. Tools like Dynatrace, Amazon Forecast and Akamas, among others, are leading the charge in AI-fueled performance management, streamlining processes and delivering superior results.

    Yet, as promising as the future of AI in performance management appears, organizations must navigate challenges to fully harness its potential. From managing data quality to bridging the skills gap, countering resistance to change, integrating with existing systems, managing costs and ensuring security and privacy, the transformation involves overcoming significant hurdles.

    However, with robust data governance, investment in upskilling, open communication, careful planning and strict adherence to security best practices, organizations can fully leverage AI to revolutionize their performance management practices and elevate their SRE capabilities. This not only ensures optimal system performance but also positions the organization well for future growth and innovation.

  • Automation or Concatenation?

    Automation or Concatenation?

    We’ve been in a period of mergers and acquisitions for a while in the DevOps and DevSecOps spaces, and it seems to be heating up even more. Combined with products expanding to cover a more broad segment of the software development life cycle (SDLC) and strategic partnerships to cover more of the SDLC, it is starting to look like our automation efforts are going to be increasingly driven by vendors. Those efforts will be more concatenation than automation–though technically the same in perception, they vary wildly in implementation, and we should be planning for it.

    Build Versus Buy

    We have always struggled with build versus buy, and we have always struggled with best-of-breed versus overarching solution set. The entire hype of Every. Single. Vendor. Claiming to be a platform is partially driven by the latter argument. Putting “platform” in your marketing does not make a vendor better at those things outside its historical strengths, though, so that movement is doomed to fail for most organizations. Some were already moving to be a platform and had assembled–by M&A or by internal development–an impressive platform already, and those were successful because they were good at a breadth of things. The best of them are continuing this trend. I try to avoid plugging vendors here, but I think an example is in order in this case.

    Akamai’s recent purchase of NeoSec is the poster child for vendors doing the platform thing super well but not resting on their laurels. I’ve followed Akamai’s security offerings for years, and the platform has impressive functionality, but NeoSec is exactly what they needed. It plugs a couple of holes in their security offerings and brings a whole customer base into the Akamai fold. That’s what I’m talking about when I refer to the good ones.

    The problem is the acquisition and expansion of offerings trend is starting to gain a lot of steam, and not every vendor jumping on the bandwagon does the job as well as Akamai does.

    This leaves enterprise IT staff responsible for DevOps and DevSecOps – even straight security staff – with a new set of issues. Best-of-breed is taking a hit again, and overarching solutions are on the rise. The difference is the breadth that some of these new “platforms” offer is massive. And we know that even the good ones will have better and worse parts of the overall solution. So IT now has to start considering what vendors call a feature and how it individually compares to the competition. This will require more up-front work for evaluators. Now, a larger (and growing) list of “features” that were once “products” has to be considered piece-by-piece against the competition, and organizations must evaluate the feasibility (and cost) of implementing two (or even multiple) massive platforms to get the most critical functionality needed.

    As of this writing, point solutions that do one thing extremely well are holding up, but the trend is obvious. Those markets will be subsumed into larger offerings that may not do as well at other parts of the overall problem. Evaluators will have to be more careful and consider their options with an eye to the larger infrastructure and problem domain. Do you want a single vendor doing all of the steps of DevOps or responsible for every aspect of application security? That will be an individual decision and, if not, how the final solution/architecture looks will be a more complex discussion.

    Integration and Automation

    In our quest for integration and automation, many vendors are responding with the concatenation of tools to solve the automation part. You’ll be stuck figuring out if they’re doing it well enough for the org.

    But that’s not a problem you can’t resolve. You’ve been rocking it through staffing issues, lockdowns, growth and shrinking… Just keep rocking it. Choose the solution that makes the overall best sense for your organization, get it running and take a break; head to the beach for some relaxation time. Or whatever you do to clear your head – because the next problem will still be there when you get back, and it’s better to be relaxed when you tackle it.

  • Harnessing AI for Automated and Toil-Free SRE

    Harnessing AI for Automated and Toil-Free SRE

    In my recent blog Revolutionizing the Nine Pillars of SRE With AI-Engineered Tools, I indicated that AI can help reduce toil by automating more complex tasks that were previously challenging to automate. Machine learning models can predict system behavior, enabling proactive responses. In this blog, I explain in more detail how AI-engineered tools can be used to improve toil reduction and automation.

    AI Use Cases for Toil Reduction and Automation

    In each of the following use cases, AI not only reduces toil but also contributes to improving system reliability, efficiency and scalability, forming a critical part of modern site reliability engineering (SRE) practices.

    Anomaly Detection and Predictive Maintenance: AI can monitor system logs and operational metrics to detect anomalies, predict system failures and schedule maintenance. This leads to fewer unexpected issues and lower downtime. Tools such as Splunk‘s Machine Learning Toolkit and Elastic’s Machine Learning features can be used for this purpose.

    Automated Testing: AI can be used to automate testing procedures, ensuring code reliability before deployment and reducing the toil of manual testing. Tools like Mabl or Testim use machine learning to create, execute and maintain tests, reducing the need for human intervention.

    Incident Triage and Resolution: AI can analyze incident data, diagnose issues, suggest solutions and, in some cases, automate the resolution process. Tools such as PagerDuty’s Event Intelligence or BigPanda help automate these processes and reduce resolution times.

    Capacity Planning and Resource Allocation: AI can predict system load and adjust resources accordingly to ensure optimal performance and scalability. Dynatrace and Datadog are examples of AI-driven platforms that can provide predictive analytics and capacity planning.

    Proactive Fault Isolation: AI can help isolate faults before they escalate into significant issues, reducing downtime and maintaining system reliability. IBM’s Watson AIOps and Moogsoft’s AIOps platform use AI to quickly isolate and remediate faults.

    Security and Threat Detection: AI can monitor system activity, detect threats and prevent security incidents, reducing the toil of manual monitoring and intervention. Tools like Darktrace and CrowdStrike Falcon use AI for cybersecurity.

    Network Optimization: AI can analyze network traffic and performance data to optimize network configurations and improve system performance. Tools such as Mist (from Juniper Networks) and Cisco’s DNA Center can provide these functionalities.

    Code Review and Technical Debt Management: AI can analyze code changes and identify issues and potential technical debt, providing guidance for incremental code improvements. Tools like DeepCode and CodeClimate can help manage and reduce technical debt.

    Overcoming Challenges

    Organizations must overcome challenges with their people, processes or technologies when transforming SRE practices to using AI tools for toil reduction and automation.

    Lack of Skilled Personnel: AI technologies often require a level of expertise that may not exist within current teams. This lack of skill can be a significant barrier to the successful implementation and usage of AI tools. Invest in training existing staff and consider hiring new team members with the necessary skill sets. There are plenty of resources and courses available online to train your teams on AI tools and technologies.

    Data Quality Issues: AI tools rely on quality data to deliver reliable insights. Data might be unclean, unstructured or siloed across different systems, which can hinder the effectiveness of AI. Implement robust data management and governance practices to ensure your data is clean, structured and accessible. AI tools can also be used to assist in data cleansing and structuring processes.

    Resistance to Change: People can be resistant to new technologies, especially when it comes to something as transformative as AI. This resistance can slow down or even halt the transition to AI-based SRE practices. Develop a strong change management strategy, which could involve regular communication about the benefits and importance of the change, provide training and support and gradually integrate AI tools into existing workflows to allow for a smoother transition.

    Integration with Existing Systems: AI tools need to work in tandem with existing IT systems, tools and processes. Integration issues can prevent AI tools from accessing necessary data or functioning as expected. Plan the integration process carefully, ensuring that the selected AI tools are compatible with your existing systems. Where possible, choose AI solutions that offer flexible APIs and other integration options.

    High Costs: Implementing AI tools can require substantial investment, including purchasing or subscribing to the tools, integrating them into your existing systems and training staff. Calculate the return on investment (ROI) before implementation to understand the potential benefits and savings the AI tools can provide in the long run. Look for scalable solutions that allow you to start small and increase your investment as you see value.

    Compliance and Security Concerns: The use of AI tools, particularly those that leverage cloud computing, may raise concerns about data security and compliance with various regulations. Conduct a thorough risk assessment before implementing any AI tool. Choose vendors that adhere to industry-standard security protocols and offer comprehensive security features. Consider working with a third-party auditor to ensure compliance.

    Summary

    As organizations journey toward greater operational efficiency and reliability through SRE, the transformative potential of AI is becoming increasingly clear. AI-driven tools are revolutionizing various facets of SRE, including toil reduction, automation and defining service-level indicators, objectives and agreements. These innovative technologies can automate complex tasks, predict system behavior, refine performance targets, optimize deployments and aid in effective incident management. For example, AI tools such as Datadog, Dynatrace and New Relic can provide actionable insights into your systems, while others like Nobl9 can use machine learning to define SLOs based on historical performance data.

    However, the road to this AI-infused future is not without its challenges. Teams might grapple with issues such as a lack of skilled personnel, data quality, resistance to change, system integration, high costs and security concerns. Yet, with the right approach, these hurdles can be overcome. By investing in training and hiring skilled personnel, implementing robust data management practices, fostering a culture receptive to change and carefully planning the integration of AI tools, organizations can leverage the power of AI. Calculating ROI before implementation and choosing vendors who adhere to stringent security protocols can further ensure success. The use of AI in SRE represents a powerful convergence of technology and practice that is reshaping the landscape of modern IT operations.