devopsy.kinsta.cloud

Category: DevOps Practice

  • ScienceLogic Unveils Revamped AIOps Platform

    ScienceLogic Unveils Revamped AIOps Platform

    Via an early access program, ScienceLogic this week made available a Hollywood update to its artificial intelligence for IT operations (AIOps) platform that, among other capabilities, provides root cause analysis capabilities that can be invoked via a natural language interface.

    For the first time, this release of the ScienceLogic SL1 platform incorporates predictive and generative AI technologies the company gained with the acquisition of Zebrium in 2022.

    Michael Nappi, chief product officer for ScienceLogic, said Zebrium provided ScienceLogic with an AI model that surfaces insights in a way that is simpler for IT teams to understand and act on. In addition to recommending automated workflows to run, the platform gives IT teams the option to automatically run them when, for example, there is an IT incident that an existing workflow has been defined to address, he added. Each organization will need to determine its level of comfort in automating those processes based on the level of potential risk to the business, noted Nappi.

    In addition, the SL1 user interface displays IT operational information at the business service level to provide IT teams with a guided experience that makes it simpler to prioritize tasks based on their relevance to the business, noted Nappi. There is also now an SL1 toolkit that DevOps teams can use to build or customize PowerPacks templates for monitoring specific processes and services.

    Finally, SL1 is now integrated with Slack and WebEx to streamline collaboration across IT teams. Previously, the platform only supported Microsoft Teams.

    In effect, SL1 now provides IT teams with a cockpit through which they can invoke AI to autonomously manage a wide range of tasks, said Nappi.

    It’s not clear how much advances in AI might one day soon democratize the management of IT, but it is clear the level of expertise required to manage complex IT environments is declining. The overall goal is to reduce dependency on IT professionals, such as software engineers, who are hard to find and retain.

    The rate of change being made to complex IT environments is now also occurring faster than IT teams can track without the aid of AI, noted Nappi. ScienceLogic, in the longer term, is working toward making it possible to interact with a chatbot in real-time to enable IT teams to meet that challenge, he noted.

    Each IT organization will need to decide for itself how heavily to rely on AI to manage IT functions, but in time, many IT professionals are not going to want to work for organizations that don’t provide some level of AI to reduce the level of toil they regularly encounter today. Rather than having to manually perform a series of monotonous tasks, AI should enable IT professionals to act more like supervisors in an IT environment.

    It may be a while before that aspiration is fully realized, but in the meantime, IT teams would be well-advised to start identifying which tasks will soon be automated using AI because roles and responsibilities will evolve. The challenge and the opportunity now is determining how IT teams can add more value to the business as those transitions occur.

  • The Future of Jenkins in 2024

    The Future of Jenkins in 2024

    As software development practices evolve, the tools that support them must also adapt to stay relevant. Jenkins, the venerable automation server used by many DevOps teams, is at such a crossroads as it faces the changing landscape of cloud-native applications and Kubernetes adoption. 

    This article examines Jenkins’ current utility, particularly its features that have made it a staple in CI/CD practices, and addresses the question of whether it will maintain its relevance in 2024 and beyond.

    How Jenkins is Used by DevOps Teams

    CI and CD

    Jenkins has become an integral part of many DevOps teams due to its CI/CD capabilities. Continuous integration (CI) is a software development practice where developers regularly merge their code changes into a central repository. Continuous delivery (CD) is the practice of delivering the software in short cycles to ensure that it can be reliably released at any time.

    Jenkins supports the CI/CD approach in a number of ways. First, it allows developers to automate code integration. This means that whenever code changes are pushed to the repository, Jenkins can automatically build and test the project. Second, Jenkins allows for the continuous delivery of software by automating simple deployment processes (although it lacks full deployment orchestration capabilities).

    Build Pipeline Visualization

    Another key feature of Jenkins is its build pipeline visualization. This feature allows DevOps teams to visually represent the process of software delivery from code integration to deployment. The pipeline visualization provides a clear and intuitive representation of the software delivery process, making it easier for teams to understand and manage the process.

    Build pipeline visualization in Jenkins is customizable, allowing teams to tailor it to their specific needs. For example, it can be customized to show the status of each stage of the delivery process, from code integration to testing to deployment. This allows teams to quickly identify any issues or bottlenecks in the process and address them.

    Automated Testing

    Automated testing is another area where Jenkins shines. With Jenkins, DevOps teams can automate the process of testing their code by running scripts and integrating with automated testing tools as part of the build process.

    Jenkins supports a wide range of testing frameworks and tools, allowing teams to choose the ones that best fit their needs. Moreover, Jenkins can be configured to automatically run tests whenever changes are made to the code. This ensures that any issues or defects in the code are detected early and can be addressed promptly.

    Workflow Scripting

    Workflow scripting is another key feature of Jenkins in DevOps teams. With Jenkins, teams can script their workflows, automating the process of software delivery. This allows teams to standardize their workflows, ensuring that each delivery process is consistent and predictable.

    Jenkins supports the Groovy scripting language for workflow scripting. This means that teams can use the power and flexibility of Groovy to script their workflows. Moreover, Jenkins provides a user-friendly interface for scripting workflows, making the process easy even for those who are not familiar with Groovy.

    Current Usage and Adoption Statistics

    As of 2023, Jenkins remains a leader in the CI/CD space. With its extensive plugin architecture and large user base, it is used by thousands of organizations worldwide. According to the 2022 DevOps Survey by the Cloud Native Computing Foundation (CNCF), Jenkins was used by over 50% of respondents for automation and CI/CD, maintaining a significant presence in the market.

    According to the Jenkins website, there are over 1 million active Jenkins users and more than 200,000 active installations worldwide, with the platform handling 1 million+ jobs per day. It had about 44% of the global market share for CI/CD in 2023.

    The growth trajectory of Jenkins, however, has shown signs of plateauing when compared to newer, more cloud-native-focused CI/CD tools. This observation is backed by Google Trends data, reflecting a steady but not sharply increasing interest in Jenkins over recent years.

    Despite facing competition from emerging tools designed for cloud-native development, such as GitHub Actions, GitLab CI and CircleCI, Jenkins has responded by evolving its ecosystem. The creation of Jenkins X is a strategic move to cater to Kubernetes users and address some of the criticisms related to cloud-native development. However, as we’ll discuss, Jenkins X has not been readily adopted by the industry as of yet.

    Limitations of Jenkins in Cloud-Native Environments

    While Jenkins is a powerful tool for DevOps teams, it has serious limitations in cloud-native environments. These limitations stem from the fact that Jenkins was originally designed for traditional, monolithic applications, not for the dynamic and distributed nature of cloud-native applications.

    Statefulness

    One of the critical limitations of Jenkins in cloud-native environments is its inherent statefulness. Cloud-native architectures typically favor stateless applications, which are easier to scale and manage in dynamic environments like Kubernetes. 

    Jenkins, however, relies on maintaining state—it stores configuration, build history and user data, making it challenging to scale and manage efficiently in a cloud-native setup. This statefulness requires persistent storage, which complicates deployments in distributed systems where resiliency and horizontal scaling are key. Moreover, the risk of data loss or inconsistency increases in cloud-native environments, particularly during scaling operations or when recovering from failures.

    Configuration Complexity

    Jenkins is often criticized for its complex configuration requirements. Setting up Jenkins in a cloud environment involves intricate configurations that are not only time-consuming but also prone to errors. This complexity is exacerbated when integrating with modern cloud-native tools and services, requiring extensive customization and maintenance of plugins. 

    In addition, Jenkins’ traditional master-slave architecture was not designed for dynamic cloud environments, where resources need to be provisioned and deprovisioned rapidly. This leads to a significant operational overhead for DevOps teams.

    Immutable Infrastructure

    Adopting an immutable infrastructure approach is a cornerstone of cloud-native environments, promoting the idea of replacing or rebuilding servers rather than modifying them. 

    However, Jenkins’ traditional model does not align well with this principle. It typically requires modifications to its runtime environment for updates or changes, contrasting with the cloud-native approach of immutable deployments. This can make it challenging to maintain consistency and reliability in cloud-native systems where automation and minimal manual intervention are paramount.

    Microservices Limitations

    Jenkins was originally designed for monolithic architectures and doesn’t naturally fit into microservice-based applications, which are prevalent in cloud-native environments. In microservices architectures, applications are broken down into smaller, independently deployable services. 

    Jenkins, however, tends to treat applications as monolithic units, making it less efficient for handling multiple, independent microservices. This limitation affects its ability to provide fine-grained control over individual services’ build, test and deployment processes. Consequently, Jenkins may struggle with the high degree of automation, flexibility and scalability required in microservices environments.

    What is Jenkins X?

    Jenkins X is an open source project that provides pipeline automation, built-in GitOps, and preview environments to help teams collaborate and accelerate their software delivery at any scale. It is an improved version of the original project, designed specifically for Kubernetes, a popular container orchestration platform.

    Jenkins X streamlines the technical aspects of building, testing, and deploying software. It automates the installation of popular tools like Helm and Draft, making it easier for developers to get started. It also provides pre-configured pipelines so teams can start delivering software right out of the box.

    Jenkins X also introduces a new Jenkinsfile format, Jenkinsfile.yaml, which provides a more straightforward and human-readable way to describe pipelines. This new format, combined with the automation provided by Jenkins X, helps reduce the complexity of pipeline configuration and management in Kubernetes.

    Limitations and Criticisms

    Jenkins X has not been widely adopted by the software industry and has not achieved the respect and prominence of its predecessor, Jenkins. Let’s review the main limitations and criticisms of the tool.

    Lack of Maturity

    Jenkins X’s relatively recent entry into the market means it has not had the time to mature like its predecessor. This lack of maturity can manifest in several ways, including fewer features compared to established tools, fewer community-tested plugins, and potentially more bugs or unresolved issues. Due to the fast-paced evolution of cloud-native technologies, tools need time to adapt and stabilize; Jenkins X is still in this phase of its lifecycle.

    Moreover, a mature tool typically has extensive documentation and a large community of users who can offer support and share best practices. While Jenkins X has seen growth in its user base, it still lags behind more mature CI/CD solutions in these areas. This can lead to challenges when teams are looking to solve complex problems or when they require advanced support.

    Limited to Git Projects

    Another criticism of Jenkins X is that it’s limited to Git projects. While Git is the most popular version control system, it’s not the only one. Some teams may use other systems like Mercurial or SVN, and these teams would be unable to use Jenkins X.

    Even within the world of Git, Jenkins X has limitations. For example, it doesn’t support all Git hosting platforms. While it works with popular platforms like GitHub and Bitbucket, it doesn’t support others like GitLab. This can be a significant limitation for teams that use unsupported platforms.

    Community Support

    Finally, there are concerns about community support for Jenkins X. While the project is open source and theoretically has the potential for wide community involvement, some critics argue that it’s primary source of support is CloudBees, the company behind Jenkins.

    These critics worry that if CloudBees were to withdraw its support, the project could falter. They also argue that the project’s focus on certain tools and platforms (like Kubernetes and Git) reflects CloudBees’ business interests more than the needs of the broader DevOps community.

    Conclusion

    In conclusion, Jenkins’ role in the CI/CD landscape has been pivotal, and its continued evolution with projects like Jenkins X demonstrates an effort to adapt to the needs of modern, cloud-native environments. While Jenkins X has yet to reach the widespread adoption and maturity of its predecessor, its specialized focus on Kubernetes and GitOps reflects the shifting priorities in software development.

    Jenkins, despite showing signs of plateauing in growth, still possesses a strong user base and an established presence in the market, supported by a vast plugin ecosystem and a robust community. Its limitations in cloud-native contexts—such as managing statefulness, configuration complexity, and supporting immutable infrastructure—pose challenges that Jenkins X seeks to address.

    As we approach 2024, the relevance of Jenkins will likely hinge on its ability to integrate more seamlessly with cloud-native technologies and the willingness of the community to evolve its ecosystem. The breadth of its adoption and the depth of its integration in existing development pipelines mean Jenkins will continue to be a significant player in the DevOps toolchain. However, its long-term relevance will depend on the project’s ability to innovate and remain adaptable in a landscape that is rapidly embracing cloud-native solutions.

  • RBAC For Your CI/CD Pipeline: Why and How

    RBAC For Your CI/CD Pipeline: Why and How

    Role-based access control, or RBAC, is an approach to managing access and permissions in a system based on the roles of individual users. This concept is integral to information security and access control strategies, where specific roles are assigned to users and each role has its permissions and restrictions.

    RBAC is a powerful tool to ensure that only authorized users can access specific resources, perform certain tasks, or make changes within a system. It eliminates the need for assigning permissions to each user individually, simplifying the administration of security policies.

    In the context of a CI/CD pipeline, RBAC means providing access rights to various stages of the pipeline according to user roles. For instance, a developer may have access to the code repository and the ability to initiate builds, while a deployment manager may have permission to push changes to the production environment.

    In this article, we’ll explain why RBAC is so important for CI/CD pipelines, and provide general guidelines and best practices for implementing RBAC into your pipeline.

    Why Do You Need RBAC in Your CI/CD Pipeline? 

    The adoption of DevOps and CI/CD practices has revolutionized the way software is developed and deployed. However, this has also raised new security challenges. One of them is maintaining control over who has access to what in the CI/CD pipeline. That’s where RBAC comes in.

    Preventing Unauthorized Access

    RBAC is critical for preventing unauthorized access in the CI/CD pipeline. It ensures that only users with the appropriate roles can make changes to the code, initiate builds, or deploy to production.

    By defining specific roles and assigning them to users, it’s possible to provide fine-grained control over access to various parts of the pipeline. This can significantly reduce the risk of unauthorized users making changes that they shouldn’t, whether unintentionally or maliciously.

    Minimizing Insider Threats

    Insider threats are a significant concern for any organization. These can come from disgruntled employees, careless staff, or even hackers who’ve managed to gain access to an insider’s account.

    RBAC can help minimize the risk of insider threats by limiting what each user can do within the CI/CD pipeline. If a user’s account is compromised, RBAC ensures that the attacker can only do what the compromised user is authorized to do. In other words, they won’t have free rein over your entire pipeline.

    Ensuring Accountability and Auditability

    Another benefit of RBAC is that it can facilitate accountability and auditability. When every user action is tied to a specific role, it’s easier to track who did what, when and why.

    This can be particularly helpful in a post-incident investigation. If something goes wrong, you can quickly identify who had access to the affected parts of the pipeline and narrow down the list of potential culprits. Similarly, during an audit, you can easily demonstrate who has access to what, helping to satisfy compliance requirements.

    Compliance with Regulations

    Speaking of compliance, many industry regulations and standards require organizations to implement strict access control measures. RBAC can help you meet these requirements by providing a clear and manageable way to control access to your CI/CD pipeline.

    For instance, regulations such as the GDPR, HIPAA and PCI-DSS all require organizations to limit access to sensitive data. By implementing RBAC, you can ensure that only authorized users can access, modify, or delete sensitive data in your pipeline, helping you stay on the right side of the law.

    How to Implement an RBAC Mechanism Across Your CI/CD Pipeline

    To effectively implement role-based access control (RBAC) in your CI/CD pipeline, it’s essential to integrate an RBAC mechanism that aligns with the specific tools and technologies in use. This requires a combination of technical setup and policy enforcement. 

    Here are the key steps involved in adding RBAC to a large-scale CI/CD pipeline:

      1. Integration with identity providers (IdP): Begin by integrating your RBAC system with an existing identity provider, like LDAP, Active Directory or OAuth provider. This ensures that user identities and roles are centrally managed and consistently applied across all stages of the CI/CD pipeline.
      2. Role definition and assignment: Utilize your CI/CD platform’s native RBAC features or incorporate third-party RBAC management tools. Define roles such as developer, QA tester, security auditor, operations, etc., in the system.
      3. Refine access control: After creating the roles, define permissions at a granular level. For instance, Developers might have ‘read’ and ‘write’ access to the source code repositories but only ‘read’ access to the deployment configurations.
      4. Tool-specific configuration: For tools like Jenkins, GitLab CI or CircleCI, use their specific RBAC configuration options to set up these permissions. Leverage the APIs of your CI/CD tools to programmatically manage access, and utilize plugins or extensions that enhance RBAC capabilities.
      5. Separate environments: Physically or logically separate different environments (like development, staging, production) and apply distinct RBAC rules for each. Implement conditional access policies, where access to critical environments like production can require additional authentication or approvals.
      6. Enable temporary elevated access: Allow temporary elevation of access rights under controlled conditions, such as during incident management or critical deployments, with automatic reversion to standard privileges after a set time.
      7. Continuous monitoring and adjustment: Regularly audit access logs and monitor for unusual activities. Tools like Splunk or ELK stack can be used for monitoring and alerting on RBAC-related events.

    Best Practices for Implementing RBAC in CI/CD 

    Here are a few best practices that can help you implement RBAC more effectively and securely.

    Adopt the Principle of Least Privilege

    The principle of least privilege (PoLP) is a critical aspect of RBAC. It states that a user should be given the least amount of privileges necessary to perform their job functions. This minimizes the risk of accidental or intentional misuse of privileges.

    To implement PoLP, you should start by conducting a thorough audit of the current permissions in your CI/CD pipeline. Identify any over-privileged accounts and rectify them. Additionally, ensure that every new account is created with the least amount of privilege necessary.

    Automate RBAC Enforcement

    One of the challenges with RBAC is ensuring that the defined roles and permissions are consistently enforced. Manual enforcement can be time-consuming and prone to errors. Automating RBAC enforcement can help overcome these challenges.

    Automation can be achieved using scripts or specialized software. These tools can automatically assign roles and permissions based on predefined rules. They can also monitor the system for any deviations from the defined roles and permissions, alerting the relevant personnel when such deviations occur.

    Implement Strong Authentication Methods

    Authentication is the process of verifying the identity of a user or system. It’s the first line of defense against unauthorized access. Authentication comes one step before RBAC — first verify a user is who they say they are, and then RBAC provides the relevant authorization.

    Strong authentication methods can include multi-factor authentication, biometrics, or digital certificates. These methods provide an additional layer of security by requiring more than just a username and password to access the system.

    In addition, it’s important to ensure that the authentication process is secure. This includes encrypting sensitive data, protecting against brute-force attacks, and regularly updating and patching the authentication system.

    Maintain Comprehensive Audit Trails

    Audit trails are records of activity in the CI/CD pipeline. They provide visibility into who did what, when, and why. Audit trails can help detect unauthorized activity, investigate incidents, and demonstrate compliance with regulations. They can also provide valuable insights into the effectiveness of the RBAC system and highlight areas that may need improvement.

    Most user identity systems provide built-in audit trails of RBAC activity. In addition, you should log all user activity in the CI/CD pipeline, including successful and unsuccessful authentication attempts, changes to roles and permissions, and any deviations from the defined roles and permissions.

    Regularly Review and Update the RBAC System

    Finally, it’s important to regularly review and update the RBAC system. This includes reviewing the defined roles and permissions, the effectiveness of the enforcement mechanisms and the completeness of the audit trails.

    Regular reviews and updates can help ensure that the RBAC system remains effective and relevant. They can also help identify and address any emerging threats or vulnerabilities.

    In conclusion, implementing RBAC in your CI/CD pipeline can help ensure secure and efficient operations. By following the best practices outlined in this article, you can create a robust and effective RBAC system. Remember, the goal is not just to prevent unauthorized access but also to enable authorized access securely and efficiently.

  • Survey Surfaces Significant Platform Engineering Gains

    A survey of 100 software engineers working at organizations in the U.S. and Western Europe published today found nearly all (99%) have embraced platform engineering to some degree as a methodology to manage DevOps workflows at scale, with more than half (53%) having done so in the last year.

    Conducted by Port, a provider of a proprietary internal developer portal (IDP) platform, the survey also found 83% of respondents have also incorporated GitOps workflows into their platform engineering practices.

    The survey also found 85% are working for organizations that have either started deploying IDPs or plan to do so in the coming year. However, what constitutes an IDP is not exactly clear, with more than a third (35%) that have implemented one reporting they are relying on spreadsheets to manage their IDP. A little over half (53%) of respondents, in contrast, are using an IDP to provide developers with self-service capabilities for setting up environments to write code that either was custom built, is based on open source Backstage software originally developed by Spotify or a commercial IDP platform.

    Among the 85% of respondents using or planning to use an IDP, the main measure of success is to improve developer productivity (56%), followed by reduced time to deployment (25%). Despite those initiatives, only slightly more than a third (34%) said IDPs are helping DevOps teams enforce software engineering standards.

    Roni Floman, chief marketing officer for Port, said overall, the survey suggests developers are wasting a lot of time on tasks that have little to do with writing code. For example, the survey finds that prior to embracing platform engineering, developers spent three to four hours a day on non-core activities, resulting in weeks being required to eventually deploy software in a production environment.

    The biggest challenge organizations face when embracing platform engineering is convincing developers that a centralized team has their best interests at heart. Many application development teams embraced DevOps as a best practice in the first place to escape the tyranny of a centralized IT team. The issue is that, as a result, many of them now find themselves managing tasks that have little to do with writing code. Platform engineering seeks to strike a balance between giving developers the ability to adopt new tools when they see fit and the benefits of having a dedicated platform engineering team that automates processes on their behalf.

    Unless platform engineering teams find a way to get developers to buy into the effort, however, it’s unlikely they will succeed, noted Floman.

    Each organization will need to determine for itself what level of platform engineering makes sense for them to implement, given their unique culture. In most cases, a platform engineering initiative will start small before increasing in size and scope as momentum is gained. In the meantime, the most important thing any platform engineering team clearly needs to do in the short term is engage developers. After all, the best way to get support for any change to a DevOps workflow is, as always, to convince developers it was their idea in the first place.

  • DevOps and Databases: The Key to Faster Deployment

    DevOps and Databases: The Key to Faster Deployment

    With DevOps, we learned that the biggest factor in streamlining deployments is to make things automated and minimize communication. However, we didn’t apply that principle to other areas. While security gets some traction with DevSecOps and machine learning has MLOps, we lack a solution for database management. In this article, we propose a new approach that unlocks the power of DevOps engineers to own their databases and lets them self-serve deployments, maintenance and troubleshooting.

    Why Database Deployment and Maintenance is Slow

    In the last few years, we moved from deployment being a big and challenging event to deploying many times daily. We moved from manual deployments to automated CI/CD pipelines. We built tools for deploying infrastructure and configuration changes. This led to a tremendous increase in the complexity of our systems. We somewhat learned how to handle that in deployments, infrastructure, configuration, security or even machine learning.

    However, database complexity increased significantly as well. We now use many database engines with each application. We use SQL and NoSQL databases as well as databases tuned for specific scenarios like machine learning or vector search. Since we deploy changes often and asynchronously, modifications to database schema or background tasks may happen at any time and affect the performance. When this happens, the business is directly impacted as the database issues quickly spread throughout the ecosystem. However, developers and DevOps engineers can’t fix the issues on their own and need help from other teams, specifically operations and database administrators. This makes the process slow and error-prone as things are not automated but require manual human intervention. Recommended reading: Why DevOps should own their databases and how to do it.

    The problem with database maintenance is the lack of solutions that would present the full and actionable story. We have many tools that can alert us about spiking CPU or increased latency, but we don’t know what’s the reason behind these issues. To find out, developers need to ask database administrators to check the logs and metrics to identify slow queries and whether the CPU spikes are caused by changes in the environment (like updating the operating system), slow background tasks that can be postponed (like vacuuming or index rebuilding) or due to inefficient application code that doesn’t use an index and performs a full table scan.

    Since we don’t know the reason, we need to involve multiple teams to investigate. We need to first identify the points of contact in each team, assign tickets or work items, sometimes page people at night, and coordinate the troubleshooting process. To make things faster, we need to fix two aspects: reduce the amount of manual work and reduce the number of people involved in the investigation. Let’s see how.

    Reducing the Manual Work with Observability

    Many things can go wrong when dealing with databases. Let’s see some of the areas that can break and are not automatically fixed today.

    Observability for the Code Changes

    Every time developers change their codebase, there is a risk of breaking the correctness and performance. We can verify the correctness with automated tests (unit tests, integration tests, end-to-end tests, etc.) and do it regularly in our CI/CD pipelines. However, we can’t verify the performance impact. Our tests focus on what we extract from the database and not how we do it. We may use N+1 queries, we may scan tables instead of using indexes, and we may have slow queries with CTEs or not use searchable arguments (a query that is not sargable). All these things may result in CPU spikes or latency degradation and we can’t automatically explain what happens. We need to have observability that gives the story like “the query execution within a given REST endpoint doesn’t use an index and leads to slow performance.”

    Performance of Schema Changes

    When we modify a schema, we risk that tables will be locked for rewriting. This may take minutes or hours and effectively make our systems unavailable. However, our CI/CD pipelines do not test the performance of schema migrations. Tests run against the latest schema and migrations are fast because databases in pre-production environments are small. We don’t check how long these migrations will take in production and if they’ll lock tables. We need to have tools that can warn us if a slow migration is being deployed and will indicate how to make it faster.

    Understanding of Execution Plans

    Databases grow over time. Queries change their execution plans because statistics get outdated or because indexes are modified. However, automated tests won’t capture these issues as they focus on the correctness of the data and they don’t run on the production database. We don’t see how things evolve and that the query degrades gradually over time. We need to have observability systems that provide an understanding of all the moving pieces around. They need to give us a story explaining that “the query we deployed six months ago stopped using an index because the statistics are out of date as we injected a big batch of data last night”.

    We need a story explaining what happened, like SQL stopped using an index or SQL query is not sargable. Other reasons can be schema changes, mismatched query fields, index fragmentation or significant changes in data volume affecting the query optimizer’s decision-making.

    How to Reduce the Work

    Taking all we saw above, we need to have systems that can understand how our environment evolves and how these changes affect performance. They need to alert us when some slow changes are being deployed to production, understand how the deployments affect the performance and automatically explain the issues. Troubleshooting should be as simple as just getting the notification with links to the places that we need to fix (like configuration, a particular line of code in our application, or the cloud console) with an explanation of how to do it. Fixing should be merely approving a pull request or confirming the configuration change.

    Reducing the Number of People Involved With Self-Servicing

    Having database observability is the first step. The second step is the ownership. Developers and DevOps engineers need to own their databases the same way they own their code and deployments. They need to be able to self-serve the issues and do that without any help from operations or database administrators. This will reduce communication and make things much smoother. There are three things that we need to introduce to let developers and DevOps engineers build their ownership.

    First, we need the tools. Just like we explained in the previous section, we need tools that can explain the story. We can’t just put more responsibilities on developers and DevOps engineers without making the work simple and automated.

    Second, we need new processes. We need to have playbooks and operational procedures explaining how to use the tools, how to self-serve the maintenance, and when to escalate. Developers must be familiarized with the procedures and must know how to follow them even outside of working hours. This way we can replace some work (communication, coordination of teams, manual investigation) with some other work (self-servicing) that we can do much faster.

    Finally, we need to change the mindset. Developers and DevOps engineers need to take on new responsibilities and become true owners of their databases. Even though this sounds like more work, we can make it much less time-consuming by building good observability tools and replacing slow human communication with automated and well-defined self-servicing. It’s not about shifting the work from one place to another. It’s about making the work much smaller and much more automated so the developers and DevOps teams can do it without involving other teams. Since we reduce the cross-team collaboration, the increase in the scope of work of developers and DevOps engineers will lead to doing the work faster.

    Summary

    There are three aspects that we need to work on now. First, tools like Metis must be created. Second, Platform Engineers must use these tools to define processes. Third, developers and DevOps engineers need to take ownership. While it may seem challenging at first, we already saw with the DevOps movement that minimizing communication and owning the full scope of the system is beneficial for the business and for the engineers. We need to restructure our teams and reassign responsibilities to unlock the potential and make organizations work faster.

  • 5 Reasons to Move Beyond SRE to Observability

    5 Reasons to Move Beyond SRE to Observability

    Recently, I was asked on a podcast to explain the relationship between site reliability engineering (SRE) and observability and why I believe organizations need to move their focus from the former to the latter. Those are great questions, but to answer them accurately, I first need to point out that many organizations claim to be doing SRE or claim to be doing observability but aren’t.

    One big thing that SRE and observability have in common is that they are both often erroneously equated with monitoring.

    SRE ≠ Monitoring

    Google is credited with pioneering the SRE movement with the publication of the “Site Reliability Engineering” handbook in 2016. As the SRE movement gained momentum, organizations of all sizes pored over the handbook and began applying its principles. However, very few organizations are of the size and sophistication of Google regarding the scale of their IT operations and engineering resources.

    So, as these “normal” organizations realized how difficult it was to follow the Google SRE approach in its entirety, they often opted instead to simply apply what they could. For many, the chapter on monitoring became the focus, so much so that monitoring has become synonymous with SRE in far too many organizations today.

    Let me be clear: The role of a site reliability engineer is not to monitor alerts. The role of an SRE is to define how the engineering team should take ownership of their service. SREs are responsible for establishing a culture and creating engrained processes that are focused on the quality and reliability of infrastructure.

    SREs guide the engineering team on how an application or service should be observed and monitored, including what parameters should be looked at, what alerts they should have, and what routing path the alerts should go through. An SRE instructs on best practices, explores the tools that are being used and determines how to gather reliable data with consistency throughout the organization.

    Many organizations we work with say they want to do SRE this way, but they’re not there yet. They are still stuck on monitoring every single metric they can find.

    Observability ≠ Monitoring

    Similarly, a lot of companies I meet say that they do observability, but what they are doing is just monitoring. They monitor every single CPU of every node of every pod of every machine that is running. They have alerts for some of these, and they may even have a playbook for some of them. This is not how SRE is supposed to work, and it’s certainly not what observability is all about.

    More importantly, it’s not scalable as an organization grows to hundreds or thousands of developers and different teams that all share the same IT environment.

    Two Viewpoints: Horizontal and Vertical

    Now that we’ve established that SRE and observability are alike in that neither should be equated with monitoring, let’s contrast the two. SRE and observability are different in that they have different perspectives on the IT domain.

    SREs see the world horizontally. They conceive of their IT domain on a horizontal plane where they can observe the capacity of resources being used. They see the Kubernetes cluster, and they see the cloud utilization. And they care about ensuring that what is deployed into the production environment is secure.

    Although they see the pieces they manage on the horizontal infrastructure plane, they don’t see the ultimate impact on the application or customer on the planes “above.”

    Observability is looking at the world vertically—that is, seeing how an application is or isn’t meeting customer needs. The whole goal of observability is to know if my application is serving my customers at the service level that I agreed to. With a vertical view from application up to customer needs and business impact, I don’t care how many pods are being run, how many nodes are being deployed or what comprises the cloud infrastructure underneath my application, as long as it works.

    I do care about security insofar as I want to avoid introducing security vulnerabilities into the organization. But beyond that, all I need to know is that the application can meet the customer’s expectations and the service level objectives (SLOs) that we have established are being met, as well.

    Therein lies the heart of the matter: For a company to make the transition to observability, it needs to build the observability system around establishing and meeting SLOs.

    The Challenges of Transitioning to a Vertical Perspective

    Companies must move beyond an SRE and DevOps organization to being an observability and SLO organization. Only by doing this will organizations be capable of understanding the trade-offs within their IT environments between cost, security and availability and the ultimate impact those trade-offs will have on business objectives.

    Today, making this transition — moving beyond SRE to observability — is difficult for a wide variety of reasons, not the least of which are:

    Alert Fatigue. Companies today are monitoring too many things they shouldn’t care about, and as a result, engineers are suffering from alert fatigue. With this, you face the danger of a higher probability of missing the alerts that you do care about and making sure that you address them promptly.

    Ideally, observability solutions eliminate the irrelevant and unimportant alerts, add context to alerts that might need investigating, and correlate response data and docs to accelerate and coordinate the appropriate response of on-call responders.

    Tool sprawl. Many of the companies we work with use a wide variety of separate tools for observability. Observability requires access to the data all of these tools produce, and problems can persist if the tools operate in data silos.

    However, the problem can be solved if there is unified data collection across the toolset, ideally using OpenTelemetry. Cross-tool and cross-stack insights are particularly important for security purposes. Another issue with tool sprawl is that true observability requires that these tools, which each typically address just one aspect of observability, somehow all work together.

    Unfortunately, current systems and processes lack the “connective tissue” to offer a consolidated observability solution.

    Complexity. As we’ve seen in the results of our annual DevOps Pulse survey, the mean-time-to-resolution (MTTR) is going up instead of down, despite the number of tools that are available and the advancements made by the industry in recent years.

    Complexity is a large part of the problem, specifically concerning Kubernetes. In our 2023 DevOps Pulse Report, we found that almost 50% of respondents cited Kubernetes as their main challenge to gaining full observability into their environment.

    The adoption of Kubernetes is happening at lightning speed—even faster and higher than the adoption of cloud technology. But it’s not just the adoption of one Kubernetes—there are so many different flavors of Kubernetes, whether it’s Kubernetes running on managed services like AWS EKS or AKS or Kubernetes running on serverless with AWS Fargate. And that’s just to name a few of many, many types of Kubernetes out there.

    All of these options create flexibility, but wherever you create flexibility, you also create complexity. Complexity in the infrastructure increases the time it takes to troubleshoot and resolve issues.

    Cost. Most of us have figured out that products that include every feature under the sun are a waste of money. We don’t need expensive and extraneous clutter. Likewise, gathering and processing too much irrelevant monitoring data adds significantly to costs. Investing in observability platforms that are overloaded with bells and whistles you simply don’t need is similarly costly.

    The ideal observability platform taps AI and automation to help you easily see what’s most important to the business and respond quickly to core issues with precision and advanced insights.

    Culture change. Going from monitoring to observability is akin to making a cultural quantum leap. Observability requires a different organizational structure and assigns different responsibilities to the teams. The transition requires change management skills and patient, thoughtful execution.

    Though these challenges are significant, the observability industry is making rapid progress and organizations can expect to see solutions and technology providers that overcome these challenges shortly. So, be on the lookout for great opportunities coming down the pike.

    In the meantime, I challenge you to change your perspective from horizontal to vertical. Set your organization on the path to move beyond DevOps and SRE to true observability, where your teams will not just be alerting and chasing issues for troubleshooting but working to establish real SLOs and driving toward those to enable the strategic goals of the business.

  • KubeCon 2023: CTO.ai’s Developer Control Plane

    KubeCon 2023: CTO.ai’s Developer Control Plane

    Speaker 1: This is Techstrong TV.

    Alan Shimel: Hey, everyone, we’re back. We’re here in Chicago at KubeKon. We’re wrapping up our day two coverage, our last guest for day two. We’ll be back tomorrow, though. But let me introduce you to Kyle Campbell, right?

    Kyle Campbell: That’s right.

    Alan Shimel: Kyle is the founder of cto.ai and we’re going to find out about cto.ai and a little bit about what they’re doing here at KubeKon. But before we do that, Kyle, first of all, welcome.

    Kyle Campbell: Thank you.

    Alan Shimel: Second of all, let’s hear a little bit about Kyle.

    Kyle Campbell: Yeah, thanks.

    Alan Shimel: Tell us your kind of journey.

    Kyle Campbell: Yeah, well, first of all, great to meet you in person. I’ve talked in the past. Yeah, so my journey, I’ve told this story before, a little unconventional. I grew up in Nova Scotia in Canada, a small town. Been on the internet since the age of eight and just had no interest in the formal past. So I’ve been building software from the early days of the .com boom. I may not look at it, but I got some of the scar tissue. But I was self-taught software engineer, so open source was the key to my success and good developer tools.

    Alan Shimel: Sure.

    Kyle Campbell: And I came up through the cloud and open source era and then started founding developer platforms in 2014. The first company I built was a developer platform, the real estate space. Zillow acquired it in about eight months, which was interesting.

    Alan Shimel: Very.

    Kyle Campbell: And then I bootstrapped a DevOps agency quite successfully and started to find that there was a lot of opportunity for next generation developer platforms, which led me to cto.ai.

    Alan Shimel: Excellent, man. What a great story too. You still up in Nova Scotia?

    Kyle Campbell: I’m not. I moved to the other side of the country. I live in British Columbia now.

    Alan Shimel: Good for you.

    Kyle Campbell: Love the outdoors. Spend a lot of time with my son camping, fishing, and trying to get outdoors and just enjoy the beautiful-

    Alan Shimel: Loving it. Yeah, no, it’s beautiful. I mean, not that Nova Scotia’s not beautiful. It’s brutal in the winter, but it’s a beautiful country, part of the country.

    Let’s talk cto.ai now. So look, I’ve founded multiple companies myself. Every founder I’ve ever interviewed or spoken with in 30 years, they don’t just say wake up and say, “Oh, I feel like founding a company today.” There’s kind of like Richard Dreyfuss in Close Encounters of the Third Kind, right? There’s something driving you like, “I got to do this. This needs to get done.” What was driving you that needed to get done here with cto.ai?

    Kyle Campbell: Yeah. I mean, as I described my past, a lot of my journey was self-taught and stand on the shoulders of giants. And really important thing for me was developer experience and ease of use and tooling early on in my career because that enabled me to really drive my competencies as a developer and keep up with these people that had computer science degrees and master’s and all these things, right?

    Alan Shimel: Certainly.

    Kyle Campbell: So when I thought about that and I also thought about where the future’s going and my passion for platform engineering, it became pretty obvious to me that what was missing in the market was a set of tools that were designed to embrace the future of developer experience and how AI can impact that from code to cloud.

    So in 2016 or so, I started experimenting with the idea, and through 2020, I decided to formally launch cto.ai as a developer control plane is what we call it. The way to think about it is it’s a modern CI/CD platform where teams can compose their workflow across build, deployment, test, and preview. And as they work with these tools, they can interact with them in the CLI Slack, GitHub, talk to these workflows through sort of a prompt based model. And we collect telemetry data that allows them to then measure that against DORA metrics, which is-

    Alan Shimel: It’s standard.

    Kyle Campbell: … yeah, it’s a standard and I think, frankly, an empirical way to measure the developer experience. We don’t try to measure developer productivity. We think that’s the wrong question to ask. The question is how do I measure the experience the developer has in the same way that I would measure the experience my users have with my product? And if what we’re doing is building platforms to enable developer experience, we need a natively integrated way to measure that. And we also think that that measurement is a way to pave a future between how the developers experience those tools and how AI can support the developer experience and enhance developers with, frankly, superpowers over time.

    Alan Shimel: So I was recently at a couple of user conferences, big DevOps companies, and it was interesting. Three conferences all use the same metric, which is that developers only spend about 30% of their time developing.

    Kyle Campbell: State of DevOps Report 2018, I think?

    Alan Shimel: But it was in there and it’s been kind of verified in several others. What do you think about that? Is it good thing, bad thing, something to be proud of, something we should be working on? What’s the deal?

    Kyle Campbell: Yeah. Well, I think a pretty practical point of view in that as a software engineer, my goal is to release features that customers love and drive business value. And if I can do that at a higher velocity while maintaining quality, I mean, I think that that is a way that I, as an individual, can contribute to the success of the business and the team.

    But I think what often happens is we get really obsessed about the individual developer. And I think any developer who looks at the idea that they only spend 30% of their time adding value isn’t going to like that very much. And I think we need to reframe the conversation away from developer productivity towards developer enablement and look at the tooling and the process that the developer experience promotes and how developers go from code to cloud and ask ourselves as a business leader, as a CTO who we work with a lot, is the tooling and the workflow that we’re supporting our team with and that developer experience, what they’re experiencing, is that driving results?

    And I think DORA metrics is an empirical way to measure that, obviously, through anecdotes and surveys, but what we’ve really innovated on is creating a vertical integration with the tool chain that allows you to measure that in real time and then iterate on the tools quickly. And some of the teams can really own that themselves. And what the businesses should be doing is zooming out and asking themselves, “Yeah, how do we maximize the amount of time developers spend without individualizing that consideration?” There’s an innerloop and there’s an outerloop. And what DORA is really good for is measuring the outerloop of how we work in aggregate of teams. We need next generation platforms that make this data and this context more accessible to business leaders.

    Alan Shimel: I don’t disagree with anything you said there, so we’re there with that. What about have you guys released any news or anything? What do you got going on around the show?

    Kyle Campbell: Yeah, so for the last while, we’ve been focused on really building a rock solid workflow and tooling to basically set the premise of that next generation composable CI/CD vision. We’ve fed that data into DORA Metrics. You can now report on DORA metrics in real time, save those reports, share them with whoever one-on-ones, retros, so kind of up to you. And we’re starting to experiment with some data science, but something that’s really exciting that we just kind of put into production last week. We haven’t really officially announced, so I guess this is a first is we’ve released our first kind of AI-

    Alan Shimel: Heard it here first.

    Kyle Campbell: … yeah, heard it here first. Our first AI feature set, which is AI code review. And what we’re doing is we’re letting teams start with these sort of base models for how they can leverage AI in the code review process to get developers’ feedback earlier. But then, we want them to be able to do is enhance that model to really customize it to their preferences on their own kind of hardware, to customize those models to take into consideration their own specialized requirements and obviously protect proprietary information that’s so core to their beliefs. So we launched that into production, announcing it here first.

    Alan Shimel: Really?

    Kyle Campbell: And we’re looking for early adopters who can kind of test it out and help us stress test it and figure out how do we further enhance the context of that model with some of the information that is derived in these DORA metrics as well.

    Alan Shimel: So here, we’re talking to this camera. Developers out there. You’re looking for them to kick this around a little bit. Kick the tires.

    Kyle Campbell: That’s right. Yeah.

    Alan Shimel: How do they do it?

    Kyle Campbell: You’re a developer and you’re interested in how AI can support your CICD workflow, specifically code review, come over to cto.ai, sign up, send a message. I’d love to talk to you personally, software development yourself, and love to hear what you think, what you think are the limitations and hesitations, and more importantly, how do you think we can enhance this with other use cases? Because I think there’s a whole exciting set of use cases for AI and software delivery that is going to help software developers.

    Alan Shimel: Absolutely.

    Kyle Campbell: We embrace-

    Alan Shimel: I had this conversation in an earlier interview today. Look, for people who aren’t tech, you show them an AI interface and some of the kind of parlor tricks it could do and they think it’s magical.

    Kyle Campbell: Yes. Arthur C. Clark.

    Alan Shimel: Right.

    Kyle Campbell: Yeah.

    Alan Shimel: Now, we know it’s not magic. It’s just putting one word in front of the other or one… but there are some uses that AI really shine certain. So it’s not meant to replace Google Search nor is it meant to replace software developers.

    Kyle Campbell: No.

    Alan Shimel: But it can empower software developers to be 10x more effective. And one of the ways it could teach you to code, right? You could put code in there and say, “Explain it to me.” It could review code, right? So either it’s writing code or it’s reviewing code. You shouldn’t have it doing both probably, but…

    Kyle Campbell: It’s true.

    Alan Shimel: Right. But if it’s using it for review, what a great use of the technology.

    Kyle Campbell: Yeah. We’ve seen that the number one thing that slows down time to market in feature sets for developers is waiting for code review. And while I’m not sure that it’ll be the same thing as you and I pair programming and getting into the nitty-gritty details, if we can make sure that every developer can get a code review instantly just by adding a label and then that we can tailor the quality of that code review with more context that is supported by how the team thinks about quality. I think this is an important example of how we can accelerate the needs of developers and the developer experience with AI that still leaves all the control in the hands of the developer to decide, “Is this rational? Is this a hallucination? Is this the appropriate time?” But the feedback loop that we can get from this is, I think, a really great beachhead and why we chose to start there.

    Alan Shimel: Absolutely. And we’ll see. I don’t think we scratched the surface yet on how we’re going to use AI, but this is certainly a real definitive first step that I see us going with, right?

    Kyle Campbell: I think so. And I think the call to action for other software vendors out there is to think about reinstrumenting what they’re offering to prepare for this because I think there’s a necessary hesitance around AI. I think rightly so. But if we can think about how we can take control of it, how softwares can take control of it. Software engineers are perfectly suited to be some of the first stewards of these problems. Just like we see AI and code generation, I think the next natural step is to support it within code review. And then we’ll start looking at how smarter deployments, smarter rollbacks, and obviously, operations can be enhanced with AI. And this place is a perfect example of people trying to solve that problem.

    Alan Shimel: Absolutely. Absolutely. And here’s the other thing. I’ve been in technology a long time. You don’t put these genies back in their bottles.

    Kyle Campbell: Right.

    Alan Shimel: Once they’re out, they’re out. And so like it or not, they’re here. And I think you could… if you can’t beat them, join them.

    Kyle Campbell: You can’t stop Moore’s Law. I mean, this the pathway that the world is on.

    Alan Shimel: No, you’re not. You’re just not going to stop it. That’s the way it is. It’s great stuff, man. I love the attitude. I love having you on here. Cto.ai. Again, if you’re looking to help Kyle out, kick the tires on this AI code review product they got coming on, it’s fantastic. So let’s check it out.

    Kyle Campbell: All right. Thank you so much.

    Alan Shimel: All right. Hey, Kyle, enjoy the rest of your KubeKon. cto.ai wrapping up day two in Chicago at KubeKon. We’re going to be here tomorrow morning. I think we’re on at 10. We’re only here till two tomorrow because it wraps early tomorrow. But we’ll be back with a full load. We hope you’ve enjoyed today. On Demand is available at Techstrong TV. And so if you missed anything, do check it out. This is Alan Shimel. We’re out.

  • Atlassian Delivers on Generative AI Promise for DevOps and ITSM

    Atlassian Delivers on Generative AI Promise for DevOps and ITSM

    During a virtual Unleash event, Atlassian today announced its generative artificial intelligence (AI) capabilities are now generally available across its Jira and Confluence suite of tools for managing IT and DevOps workflows.

    In addition, Atlassian also committed to adding generative AI capabilities to its Bitbucket continuous integration/continuous deployment (CI/CD) platform to enable DevOps teams to review pull requests and leave comments automatically with suggested changes around syntax and code conventions via a natural language interface. Developers will also be able to autogenerate pull request descriptions from commit messages.

    Matt Schvimmer, head of products for the Agile and DevOps division of Atlassian, said the capabilities provided by Atlassian Intelligence will reduce the level of toil and stress DevOps teams experience at a time when the velocity at which applications are being developed and deployed only continues to increase.

    Atlassian reported nearly 10% of its more than 265,000 customers have already leveraged Atlassian Intelligence since the launch of the beta program. Organizations, for example, can instantly create user stories within Jira Software tickets, change the tone of a customer response within Jira Service Management, create summaries, launch workflow requests via prompts and generate a starting point for a test plan in Confluence.

    Natural language capabilities, currently available for Confluence, will shortly be generally made available for Jira. An ability that promises to help demystify company-specific concepts, jargon or acronyms is also available now in beta, with support for Jira Software and Jira Service Management to shortly follow.

    There is little doubt at this juncture that AI will be pervasively applied across both ITSM and DevOps workflows. As those advances are made, it should also become feasible to reduce the level of toil these teams regularly encounter. At the same time, the level of scale at which ITSM and DevOps workflows can be effectively managed should also increase substantially.

    Each organization will need to decide how quickly to incorporate AI into those workflows, but hopefully, the level of burnout experienced by IT personnel will be sharply reduced as more tasks are automated. Less clear is the impact AI will have on the size of IT teams required to provide those services, but for the foreseeable future, there will always be a need for some level of human supervision.

    In the longer term, the war for IT and DevOps talent that has been waged now for multiple decades should diminish as managing these workflows becomes more accessible, noted Schvimmer. Organizations will not need to rely so much on the expertise of a handful of specialists, he added.

    Organizations, however, would be well-advised to make sure AI is embraced from both the bottom up and the top down to ensure optimal adoption, said Schvimmer.

    In the meantime, IT teams should take an inventory of the processes that are likely to be automated by AI today with an eye toward restructuring teams as more tasks are automated. Ultimately, the goal should be to let machines handle the tasks they do best to enable the IT staff to provide services that deliver more value to the business.