devopsy.kinsta.cloud

Tag: devops

  • From Machine Learning to DevSecOps: Six DevOps Trends for 2024

    From Machine Learning to DevSecOps: Six DevOps Trends for 2024

    The DevOps market is set to grow by 25% annually between 2024 and 2032, fueled by the growing demand for faster software delivery and higher-quality output. Emerging technologies like AI-driven development and serverless architectures have introduced new products in the space, while traditional processes like version control and CI/CD will be further refined. But amongst this innovation, security and quality should remain as organizations’ highest concerns. Looking into the coming year, it’s hard not to see where the DevOps industry is going based on the past year. As such, we have endeavored to provide six distinct DevOps trends we think will be particularly influential in the new year.

    The Integration of AI and Machine Learning

    One of the most notable trends in the future of DevOps is the increasing incorporation of artificial intelligence (AI) and machine learning (ML) into the software development life cycle. The explosion of AI technology in 2023 has set the stage for more predictive analytics in code development, automated testing and intelligent monitoring in the operations phase.

    As Google’s 2023 State of DevOps Report highlighted, DevOps teams feel that AI will contribute most to analyzing data and security, alongside identifying bugs. However, a solid use case has yet to materialize for most teams, and the report shows that early adopters of AI have experienced a minor decrease in their software delivery performance and a significant decrease in their operational performance. So, it’s possible that at this stage teams are implementing AI without considering what problems they’re trying to solve, or without paying full attention to the security and quality implications.

    Emphasis on Increased Productivity

    While AI adoption will be driven by the underlying push for increased productivity in DevOps tools and practices, DevOps teams should focus on drawing maximum value from the existing tools that automate mundane and repetitive tasks, allowing them to focus on more critical aspects of development and operations. This will be made possible by DevOps vendors that provide enhancements to collaborative tools, simplified interfaces and more intuitive workflows, reducing the cognitive load on developers and operations professionals.

    Culturally, the continuing trend towards remote and hybrid work environments is influencing productivity strategies in DevOps, with tools and practices that support remote collaboration and communication becoming vital. Cloud-based DevOps tools are facilitating seamless collaboration, allowing teams to work efficiently irrespective of their physical locations.

    Alignment to Existing Cloud and Microservices Architecture

    We can expect the new alignment of DevOps with cloud and microservices architectures to continue into 2024. This integration is about more than just adopting new technologies — it’s about creating efficiencies that enhance agility and revenue growth.

    The combination of DevOps practices with cloud and microservices enables organizations to leverage the inherent benefits of these platforms. Cloud infrastructure is scalable and flexible, reducing overhead costs associated with physical servers. Microservices allow for rapid development, testing and deployment of independent components without disrupting the whole system. Together with DevOps automation and continuous practices, these architectures accelerate innovation and time-to-market.

    This alignment of DevOps with cloud and microservices will lead to enhanced scalability, resilience, collaboration and customer experiences. By being able to quickly scale, update, and respond to demands, organizations can better satisfy customers and meet market needs, and this, in turn, will drive revenue growth and productivity.

    The Drive for Higher Quality

    While increased productivity and revenue generation will be a key priority in 2024, it’ll be matched by an unwavering commitment to quality. With the rising complexity of software and the increasing dependency of businesses on digital platforms, the margin for error is slimming. Quality assurance practices will become more integrated into the DevOps workflow, ensuring that each release meets the highest standards of performance, security, and user experience.

    Continuous integration/continuous deployment (CI/CD) practices are evolving to include more rigorous and automated testing protocols. These protocols are designed not just to catch bugs but to assess performance under diverse and real-world scenarios. Real-time monitoring tools are being integrated more deeply into DevOps workflows, providing ongoing insights into application performance and user experience. This continuous feedback loop ensures that any issues are identified and addressed promptly, thereby maintaining high standards of quality post-deployment.

    DevSecOps Maturity

    In the same vein, cybersecurity will continue to be a primary concern in 2024, with core DevOps tooling and processes stepping up by incorporating advanced security features. Version control systems will take the lead here by introducing robust access controls, encryption of code repositories and integration with security testing tools to ensure that the code is not only efficient but also secure from vulnerabilities.

    Security integrations in version control will become more advanced, possibly incorporating real-time scanning for security vulnerabilities, tighter integrations with identity management platforms and pattern analysis to predict and prevent potential security breaches in code.

    And as AI is pioneered throughout DevOps tooling, organizations will need to consider the security implications. The Open Worldwide Application Security Project (OWASP) has predicted that data leaks, permission issues and insecure plugins could pose serious security vulnerabilities for software. Teams that can harness growth with AI in 2024 while building sufficient security through DevSecOps practices like backup and monitoring will future-proof themselves.

    Speed and Security

    The speed of DevOps growth, innovation and adoption is only set to increase in 2024. We’ll see teams pioneer new technologies to meet business priorities and revenue targets. In doing so, it’s very easy to jump on the AI or serverless bandwagon and assume that sprinkling it on every technical discipline will make for a bright and shiny future. While some key opportunities in the DevOps space will benefit from the acceleration and tighter integration that new technology brings, organizations looking for lasting growth will need to prioritize quality and security practices.

    The key trend we’ll see in 2024 is teams getting smarter and more focused on how they apply new technology to their DevOps trajectory, delivering faster, more accurate releases that free up time to deliver ever greater value to their customers and business. So, the DevOps winners of 2024 will be the organizations that manage to balance the speed of new technology adoption with robust security.

  • Opsera Leverages AI to Simplify DevOps Platform Migrations

    Opsera Leverages AI to Simplify DevOps Platform Migrations

    Opsera is leveraging a generative artificial intelligence (AI) model it built to enable DevOps teams to migrate from one platform to another.

    Gregg Holtzrichter, chief marketing officer for Opsera, said the Hummingbird AI model makes it simpler to migrate source code from, for example, a Bitbucket on-premises continuous integration/continuous development (CI/CD) platform to GitHub.

    Opsera originally developed Hummingbird as an orchestration engine that integrates DevOps workflows spanning multiple tools and platforms. The primary goal is to make it possible for DevOps teams to unify their workflows, said Holtzrichter.

    At the same time, Opsera is now making it possible to leverage a generative AI model to reverse engineer the source code a DevOps team may have developed to extend the capabilities of a specific platform. That new code then makes it possible to significantly reduce the total cost of switching DevOps platforms, said Holtzrichter. As part of such an effort, Opsera is also now providing support for organizations looking to specifically migrate to GitHub Copilot and GitHub Actions.

    It’s not clear how many DevOps teams might be looking to switch platforms, but with the rise of generative AI capabilities for writing code, interest in platforms such as GitHub has increased, noted Holtzrichter. In addition, support for platforms such as the on-premises edition of Bitbucket is being sunsetted by Atlassian as part of an effort to drive more customers toward the cloud edition of the CI/CD platform.

    Regardless of the underlying platforms involved, generative AI promises to reduce the ability of any vendor to lock customers into a specific platform. As DevOps teams extended DevOps platforms over the past decade, many of them have discovered that the integrations they created and maintained made it all but impossible to switch platforms. With the ability to refactor much of that code based on the insights surfaced by the Opsera platform using generative AI, the time and effort required to switch platforms has been substantially reduced, noted Holtzrichter.

    At its core, Opsera makes it possible to apply analytics to DevOps workflows as they become more complex. The number of dependencies that exist between various application modules, for example, makes it exceedingly difficult for DevOps teams to manually identify the impact a delay in one project might have on any number of other downstream projects.

    The Opsera platform surfaces that intelligence by scanning all repositories and branches, including commit history and generating alerts and notifications whenever KPI thresholds or compliance requirements are exceeded. DevOps teams can also use the platform to assess their overall DevOps maturity using, for example, the DevOps Research and Assessment (DORA) metrics defined by Google. In addition, many organizations are keen to discover what best practices should be adopted by other application development teams.

    Now that AI is being applied to that analytics capability, it’s become possible to extend the reach of the Ospera platform beyond analytics to include source code migration. Each organization will still need to determine whether such an effort is worth the remaining time and effort still required, but at the very least, most DevOps teams can take comfort in the fact they can now keep their options a lot more open.

  • What Will DevOps Look Like in 2024?

    What Will DevOps Look Like in 2024?

    The last two years have been marked by significant downsizing in technology companies. As large layoffs at companies like Amazon, Meta and Google make major headlines, there is a general shift away from the “spending cash and growing fast” mindset that free capital and strong growth during the pandemic launched in the tech industry. The name of the game is to be more conservative with spending, and that mindset will drive a lot of investments in the next year. While these cost pressures may feel challenging to technology organizations, there are a few key opportunities as well. Many trends are starting, sustaining and ending, so let’s focus on the top six likely to drive changes in the industry in 2024 and beyond.

    1. AI—Specifically Generative AI—Move Right on the Hype Cycle

    While AI is not new, OpenAI and ChatGPT have created an amazingly accessible and powerful tool that is positioned to drive a lot of value. The hype cycle is still in full swing as dozens of new uses of the technology hit the market daily, and if you’re not trying to build a generative AI-enabled product, you’re getting asked when you will start. In 2024, we will likely see LLMs move into the trough of disillusionment as inflated expectations and projects don’t all play out. However, those organizations with generative AI products that truly deliver value to users should expect to see massive growth. Technology organizations will need to make hard decisions about AI products. You will either find traction in the next year and see huge growth or quickly find it doesn’t work and shift focus back to non-AI products.

    2. Developer Experience Gains Momentum

    While DX has been talked about for years, we’re seeing new investments in tools that empower developers. This trend is partially born out of a need to reduce tool and system sprawl for engineering teams, to empower teams and allow them to move faster and to improve efficiency. While you may not build or buy a DX platform, you will likely start a DX team or committee that better represents the folks building your software. This will lead to some great internal changes and key efficiencies getting attention that may have been previously overlooked. The wonderful thing about this trend is that it both brings efficiencies to the organization and makes developers’ lives easier. It’s a win-win!

    3. Increased Pressure to Reduce Costs

    Even for organizations that have avoided layoffs, most are seeing pressure to shift into a leaner existence. This often means critically looking at all sources of spending and taking on new projects to optimize them. Maybe a cloud re-platform can reduce hosting spend by 20%, or maybe you start looking to remove or combine SaaS tools to reduce seats. You may even see hiring freezes for critical teams. Organizations will have to embrace other trends to increase the efficiencies of their teams to meet business demands here. Don’t be surprised to hear “do more with less” become a mantra around the office.

    4. Increased Accountability Through Data

    While data platforms for technology organizations continue to grow, teams are finding that it’s not enough to just measure a few metrics. Organizations will be required increasingly to use the data they have to show they are delivering value. Partially due to increased pressures mentioned above, teams will need to become more predictable, and better at identifying and handling risk at every stage of development. It’s no longer enough to just build a great development engine, teams must ensure that the engine can power them to their destination.

    5. ESG in Technology

    Focus on DEI in technology continues to require significant investment, however, organizations are starting to consider a broader umbrella of social responsibility. This can range from considering the environmental impacts or carbon production of your servers to changing your software supply chain to avoid supporting particular organizations. These types of requirements are appearing sooner and more overtly in the software development process, and may significantly impact how teams choose to build or implement software.

    6. DevSecOps Continues to Grow

    There has been a renewed focus on security in recent years after some critical attacks exposed vulnerabilities in the software supply chain. It’s not surprising that security is now written into the fabric of how we build and deliver software. Teams are now expected to consider security at all stages of development from product planning to operation in production, and require additional experts, training and resources to both monitor and ensure compliance. Requirements for SaaS products continue to evolve as the last remnants of on-premises solutions evaporate, and teams should be prepared to move quickly as new threats are discovered and need to be patched.

    As the technology industry prepares for 2024, it’s clear that innovation and adaptation will be the keys to success. From the continued evolution of AI and the growing focus on the developer experience to the imperative to reduce costs, increase accountability through data, embrace ESG practices  and fortify security through DevSecOps, technology organizations face an exciting future.

  • Five Great DevOps Job Opportunities

    Five Great DevOps Job Opportunities

    DevOps.com is now providing a weekly DevOps jobs report through which opportunities for DevOps professionals will be highlighted to better serve our audience.

    Our goal in these challenging economic times is to make it easier for DevOps professionals to advance their careers.

    Of course, the pool of available DevOps talent is still relatively constrained, so when one DevOps professional takes on a new role it tends to create an opportunity for others.

    The five job postings shared this week are selected based on the company looking to hire, the vertical industry segment and, naturally, the pay scale being offered.

    We’re also committed to providing additional insights into the state of the DevOps job market. In the meantime, for your consideration:

    Indeed.com

    MedStudy
    Remote
    Senior DevOps Engineer
    $125,000 to $165,000

    SimplyHired.com

    Raytheon
    Tewksbury, Massachusetts
    Technical Fellow – DevSecOps/Cloud Native
    $122,000 to $155,000

    CareerBuilder.com

    State Street Corp.
    Quincy, Massachusetts
    Azure DevOps Engineer
    $120,000 to $202,500

    Dice.com

    USEReady
    New York, New York
    Senior Azure DevOps Engineer
    $120,000 to $160,000

    LinkedIn

    Northrop Grumman
    Dulles, Virginia
    Senior Principal DevOps Engineer
    $107,300 to $199,600

  • RBAC For Your CI/CD Pipeline: Why and How

    RBAC For Your CI/CD Pipeline: Why and How

    Role-based access control, or RBAC, is an approach to managing access and permissions in a system based on the roles of individual users. This concept is integral to information security and access control strategies, where specific roles are assigned to users and each role has its permissions and restrictions.

    RBAC is a powerful tool to ensure that only authorized users can access specific resources, perform certain tasks, or make changes within a system. It eliminates the need for assigning permissions to each user individually, simplifying the administration of security policies.

    In the context of a CI/CD pipeline, RBAC means providing access rights to various stages of the pipeline according to user roles. For instance, a developer may have access to the code repository and the ability to initiate builds, while a deployment manager may have permission to push changes to the production environment.

    In this article, we’ll explain why RBAC is so important for CI/CD pipelines, and provide general guidelines and best practices for implementing RBAC into your pipeline.

    Why Do You Need RBAC in Your CI/CD Pipeline? 

    The adoption of DevOps and CI/CD practices has revolutionized the way software is developed and deployed. However, this has also raised new security challenges. One of them is maintaining control over who has access to what in the CI/CD pipeline. That’s where RBAC comes in.

    Preventing Unauthorized Access

    RBAC is critical for preventing unauthorized access in the CI/CD pipeline. It ensures that only users with the appropriate roles can make changes to the code, initiate builds, or deploy to production.

    By defining specific roles and assigning them to users, it’s possible to provide fine-grained control over access to various parts of the pipeline. This can significantly reduce the risk of unauthorized users making changes that they shouldn’t, whether unintentionally or maliciously.

    Minimizing Insider Threats

    Insider threats are a significant concern for any organization. These can come from disgruntled employees, careless staff, or even hackers who’ve managed to gain access to an insider’s account.

    RBAC can help minimize the risk of insider threats by limiting what each user can do within the CI/CD pipeline. If a user’s account is compromised, RBAC ensures that the attacker can only do what the compromised user is authorized to do. In other words, they won’t have free rein over your entire pipeline.

    Ensuring Accountability and Auditability

    Another benefit of RBAC is that it can facilitate accountability and auditability. When every user action is tied to a specific role, it’s easier to track who did what, when and why.

    This can be particularly helpful in a post-incident investigation. If something goes wrong, you can quickly identify who had access to the affected parts of the pipeline and narrow down the list of potential culprits. Similarly, during an audit, you can easily demonstrate who has access to what, helping to satisfy compliance requirements.

    Compliance with Regulations

    Speaking of compliance, many industry regulations and standards require organizations to implement strict access control measures. RBAC can help you meet these requirements by providing a clear and manageable way to control access to your CI/CD pipeline.

    For instance, regulations such as the GDPR, HIPAA and PCI-DSS all require organizations to limit access to sensitive data. By implementing RBAC, you can ensure that only authorized users can access, modify, or delete sensitive data in your pipeline, helping you stay on the right side of the law.

    How to Implement an RBAC Mechanism Across Your CI/CD Pipeline

    To effectively implement role-based access control (RBAC) in your CI/CD pipeline, it’s essential to integrate an RBAC mechanism that aligns with the specific tools and technologies in use. This requires a combination of technical setup and policy enforcement. 

    Here are the key steps involved in adding RBAC to a large-scale CI/CD pipeline:

      1. Integration with identity providers (IdP): Begin by integrating your RBAC system with an existing identity provider, like LDAP, Active Directory or OAuth provider. This ensures that user identities and roles are centrally managed and consistently applied across all stages of the CI/CD pipeline.
      2. Role definition and assignment: Utilize your CI/CD platform’s native RBAC features or incorporate third-party RBAC management tools. Define roles such as developer, QA tester, security auditor, operations, etc., in the system.
      3. Refine access control: After creating the roles, define permissions at a granular level. For instance, Developers might have ‘read’ and ‘write’ access to the source code repositories but only ‘read’ access to the deployment configurations.
      4. Tool-specific configuration: For tools like Jenkins, GitLab CI or CircleCI, use their specific RBAC configuration options to set up these permissions. Leverage the APIs of your CI/CD tools to programmatically manage access, and utilize plugins or extensions that enhance RBAC capabilities.
      5. Separate environments: Physically or logically separate different environments (like development, staging, production) and apply distinct RBAC rules for each. Implement conditional access policies, where access to critical environments like production can require additional authentication or approvals.
      6. Enable temporary elevated access: Allow temporary elevation of access rights under controlled conditions, such as during incident management or critical deployments, with automatic reversion to standard privileges after a set time.
      7. Continuous monitoring and adjustment: Regularly audit access logs and monitor for unusual activities. Tools like Splunk or ELK stack can be used for monitoring and alerting on RBAC-related events.

    Best Practices for Implementing RBAC in CI/CD 

    Here are a few best practices that can help you implement RBAC more effectively and securely.

    Adopt the Principle of Least Privilege

    The principle of least privilege (PoLP) is a critical aspect of RBAC. It states that a user should be given the least amount of privileges necessary to perform their job functions. This minimizes the risk of accidental or intentional misuse of privileges.

    To implement PoLP, you should start by conducting a thorough audit of the current permissions in your CI/CD pipeline. Identify any over-privileged accounts and rectify them. Additionally, ensure that every new account is created with the least amount of privilege necessary.

    Automate RBAC Enforcement

    One of the challenges with RBAC is ensuring that the defined roles and permissions are consistently enforced. Manual enforcement can be time-consuming and prone to errors. Automating RBAC enforcement can help overcome these challenges.

    Automation can be achieved using scripts or specialized software. These tools can automatically assign roles and permissions based on predefined rules. They can also monitor the system for any deviations from the defined roles and permissions, alerting the relevant personnel when such deviations occur.

    Implement Strong Authentication Methods

    Authentication is the process of verifying the identity of a user or system. It’s the first line of defense against unauthorized access. Authentication comes one step before RBAC — first verify a user is who they say they are, and then RBAC provides the relevant authorization.

    Strong authentication methods can include multi-factor authentication, biometrics, or digital certificates. These methods provide an additional layer of security by requiring more than just a username and password to access the system.

    In addition, it’s important to ensure that the authentication process is secure. This includes encrypting sensitive data, protecting against brute-force attacks, and regularly updating and patching the authentication system.

    Maintain Comprehensive Audit Trails

    Audit trails are records of activity in the CI/CD pipeline. They provide visibility into who did what, when, and why. Audit trails can help detect unauthorized activity, investigate incidents, and demonstrate compliance with regulations. They can also provide valuable insights into the effectiveness of the RBAC system and highlight areas that may need improvement.

    Most user identity systems provide built-in audit trails of RBAC activity. In addition, you should log all user activity in the CI/CD pipeline, including successful and unsuccessful authentication attempts, changes to roles and permissions, and any deviations from the defined roles and permissions.

    Regularly Review and Update the RBAC System

    Finally, it’s important to regularly review and update the RBAC system. This includes reviewing the defined roles and permissions, the effectiveness of the enforcement mechanisms and the completeness of the audit trails.

    Regular reviews and updates can help ensure that the RBAC system remains effective and relevant. They can also help identify and address any emerging threats or vulnerabilities.

    In conclusion, implementing RBAC in your CI/CD pipeline can help ensure secure and efficient operations. By following the best practices outlined in this article, you can create a robust and effective RBAC system. Remember, the goal is not just to prevent unauthorized access but also to enable authorized access securely and efficiently.

  • DevOps and Databases: The Key to Faster Deployment

    DevOps and Databases: The Key to Faster Deployment

    With DevOps, we learned that the biggest factor in streamlining deployments is to make things automated and minimize communication. However, we didn’t apply that principle to other areas. While security gets some traction with DevSecOps and machine learning has MLOps, we lack a solution for database management. In this article, we propose a new approach that unlocks the power of DevOps engineers to own their databases and lets them self-serve deployments, maintenance and troubleshooting.

    Why Database Deployment and Maintenance is Slow

    In the last few years, we moved from deployment being a big and challenging event to deploying many times daily. We moved from manual deployments to automated CI/CD pipelines. We built tools for deploying infrastructure and configuration changes. This led to a tremendous increase in the complexity of our systems. We somewhat learned how to handle that in deployments, infrastructure, configuration, security or even machine learning.

    However, database complexity increased significantly as well. We now use many database engines with each application. We use SQL and NoSQL databases as well as databases tuned for specific scenarios like machine learning or vector search. Since we deploy changes often and asynchronously, modifications to database schema or background tasks may happen at any time and affect the performance. When this happens, the business is directly impacted as the database issues quickly spread throughout the ecosystem. However, developers and DevOps engineers can’t fix the issues on their own and need help from other teams, specifically operations and database administrators. This makes the process slow and error-prone as things are not automated but require manual human intervention. Recommended reading: Why DevOps should own their databases and how to do it.

    The problem with database maintenance is the lack of solutions that would present the full and actionable story. We have many tools that can alert us about spiking CPU or increased latency, but we don’t know what’s the reason behind these issues. To find out, developers need to ask database administrators to check the logs and metrics to identify slow queries and whether the CPU spikes are caused by changes in the environment (like updating the operating system), slow background tasks that can be postponed (like vacuuming or index rebuilding) or due to inefficient application code that doesn’t use an index and performs a full table scan.

    Since we don’t know the reason, we need to involve multiple teams to investigate. We need to first identify the points of contact in each team, assign tickets or work items, sometimes page people at night, and coordinate the troubleshooting process. To make things faster, we need to fix two aspects: reduce the amount of manual work and reduce the number of people involved in the investigation. Let’s see how.

    Reducing the Manual Work with Observability

    Many things can go wrong when dealing with databases. Let’s see some of the areas that can break and are not automatically fixed today.

    Observability for the Code Changes

    Every time developers change their codebase, there is a risk of breaking the correctness and performance. We can verify the correctness with automated tests (unit tests, integration tests, end-to-end tests, etc.) and do it regularly in our CI/CD pipelines. However, we can’t verify the performance impact. Our tests focus on what we extract from the database and not how we do it. We may use N+1 queries, we may scan tables instead of using indexes, and we may have slow queries with CTEs or not use searchable arguments (a query that is not sargable). All these things may result in CPU spikes or latency degradation and we can’t automatically explain what happens. We need to have observability that gives the story like “the query execution within a given REST endpoint doesn’t use an index and leads to slow performance.”

    Performance of Schema Changes

    When we modify a schema, we risk that tables will be locked for rewriting. This may take minutes or hours and effectively make our systems unavailable. However, our CI/CD pipelines do not test the performance of schema migrations. Tests run against the latest schema and migrations are fast because databases in pre-production environments are small. We don’t check how long these migrations will take in production and if they’ll lock tables. We need to have tools that can warn us if a slow migration is being deployed and will indicate how to make it faster.

    Understanding of Execution Plans

    Databases grow over time. Queries change their execution plans because statistics get outdated or because indexes are modified. However, automated tests won’t capture these issues as they focus on the correctness of the data and they don’t run on the production database. We don’t see how things evolve and that the query degrades gradually over time. We need to have observability systems that provide an understanding of all the moving pieces around. They need to give us a story explaining that “the query we deployed six months ago stopped using an index because the statistics are out of date as we injected a big batch of data last night”.

    We need a story explaining what happened, like SQL stopped using an index or SQL query is not sargable. Other reasons can be schema changes, mismatched query fields, index fragmentation or significant changes in data volume affecting the query optimizer’s decision-making.

    How to Reduce the Work

    Taking all we saw above, we need to have systems that can understand how our environment evolves and how these changes affect performance. They need to alert us when some slow changes are being deployed to production, understand how the deployments affect the performance and automatically explain the issues. Troubleshooting should be as simple as just getting the notification with links to the places that we need to fix (like configuration, a particular line of code in our application, or the cloud console) with an explanation of how to do it. Fixing should be merely approving a pull request or confirming the configuration change.

    Reducing the Number of People Involved With Self-Servicing

    Having database observability is the first step. The second step is the ownership. Developers and DevOps engineers need to own their databases the same way they own their code and deployments. They need to be able to self-serve the issues and do that without any help from operations or database administrators. This will reduce communication and make things much smoother. There are three things that we need to introduce to let developers and DevOps engineers build their ownership.

    First, we need the tools. Just like we explained in the previous section, we need tools that can explain the story. We can’t just put more responsibilities on developers and DevOps engineers without making the work simple and automated.

    Second, we need new processes. We need to have playbooks and operational procedures explaining how to use the tools, how to self-serve the maintenance, and when to escalate. Developers must be familiarized with the procedures and must know how to follow them even outside of working hours. This way we can replace some work (communication, coordination of teams, manual investigation) with some other work (self-servicing) that we can do much faster.

    Finally, we need to change the mindset. Developers and DevOps engineers need to take on new responsibilities and become true owners of their databases. Even though this sounds like more work, we can make it much less time-consuming by building good observability tools and replacing slow human communication with automated and well-defined self-servicing. It’s not about shifting the work from one place to another. It’s about making the work much smaller and much more automated so the developers and DevOps teams can do it without involving other teams. Since we reduce the cross-team collaboration, the increase in the scope of work of developers and DevOps engineers will lead to doing the work faster.

    Summary

    There are three aspects that we need to work on now. First, tools like Metis must be created. Second, Platform Engineers must use these tools to define processes. Third, developers and DevOps engineers need to take ownership. While it may seem challenging at first, we already saw with the DevOps movement that minimizing communication and owning the full scope of the system is beneficial for the business and for the engineers. We need to restructure our teams and reassign responsibilities to unlock the potential and make organizations work faster.

  • Data Observability and its Importance: Everything you Need to Know

    Data Observability and its Importance: Everything you Need to Know

    In today’s tech-driven world, companies rely on almost incomprehensible amounts of data to dictate their business operations and decisions. Enterprises are focused on building multiple data repositories and data pipelines to process, store, manage and utilize data from various sources. Given the ever-growing size and complexity of the enterprise data environment, providing the accuracy and completeness of data is becoming increasingly challenging. So how can you understand how the data is performing across the entire IT infrastructure, all at once? The answer is data observability.

    What is Data Observability?

    Data observability is a process that aims to alert you to your data’s reliability and health while also delivering the insights and analysis required to identify and resolve issues before they affect your entire organization.

    Monitoring operations in the database serves as a proactive defense mechanism against potential security threats.
    This ensures accurate, complete, secure and valuable data and eliminates data downtime. Let’s dive deeper into the observability infrastructure and explore some of the best data observability platforms that can help you optimize and secure your data operations.

    Data Observability Framework

    We know that observability has three pillars: Logs, traces and metrics.

    However, data observability has five pillars, which, together, provide key insights into the reliability and quality of your data.

    Incorporating each pillar can help you craft an effective observability strategy.

    1. Recency: Also referred to as “freshness,” recency involves confirming if the data is up to date. It also analyzes if there are unusual temporal gaps in the data tables. This helps prevent timeliness issues in data pipelines.

    2. Volume: Volume involves checking if the amount of data intake in the database meets the expected thresholds. This ensures that data sets are complete.

    3. Distribution: This measures the quality of data at the field level and confirms if the data values are within expected ranges. Unexpected fluctuations in the distribution patterns are indicative of a data issue.

    4. Schema: This involves monitoring and auditing changes to data tables and data organization to look for signs of broken data. Schemas are extremely important as changes in the source data’s structure are often the cause of data downtime.

    5. Lineage: Lineage collects metadata and provides a complete picture of your organization’s data landscape, including upstream sources, downstream, and which teams can access the data at which stages. This process helps data teams troubleshoot where breaks occur in the data.

    Data Observability Benefits

    Data observability provides better visibility into your IT system’s internal state, including its behavior, performance and interactions with other systems. This can be beneficial in the following ways:
    ● Facilitates Root Cause Analysis: With end-to-end data visibility and monitoring across multi-layered IT infrastructure, data observability enables data teams to quickly spot issues in data sets with less effort. This also increases the chances of identifying new issues regardless of where they originate.

    ● Faster Mean Time to Detection and Resolution: As data observability monitors a wide range of output, data teams can actively triage and debug their systems. By providing valuable insights into how the data interacts and moves within the IT architecture, data observability helps teams spot problems they didn’t know about, resulting in quicker mean time to detection (MTTD) and mean time to resolution (MTTR).

    ● Automates Security Management: Data observability not only provides real-time visibility into the security posture but also makes it easy to automate parts of the triage process. This helps detect data health issues or data downtime instantly.

    Data Observability Challenges

    Depending on their existing IT architecture, organizations may face the following challenges when it comes to data observability:
    ● Data Silos: Due to the presence of several agents, silos monitoring tools and disparate data sources, it becomes difficult to comprehend interdependencies among applications, digital channels and various clouds.

    ● Integration with Entire Data Ecosystem: For data observability to work, the tool requires insights into the complete data pipeline and the servers, databases, software and applications involved. However, some organizations may find it challenging to connect all of the systems to a data observability platform.

    ● Manual Instrumentation and Configuration: Data observability tools aim to standardize telemetry data and logging guidelines to efficiently correlate information. However, as large organizations maintain several data sources (hundreds or even thousands), data from these sources might have varied standards. This requires manual effort to standardize data.

    Data Observability vs. Data Governance

    In today’s digital-first landscape, where data security is a global concern, there’s a growing focus on data governance. Data governance assists in the formation of policies and procedures required to keep a check on how an organization collects, analyzes, stores, shares and uses its data.

    A strong data governance program ensures data availability, integrity, usability and security. It eliminates data integration issues, data silos and poor data quality, thereby resolving the challenges of data observability.
    With data sets now scaling with more tables, more data sources and more complexity, data engineers and developers are under pressure to keep up with the data consistency, availability and security requirements. Any amount of downtime can lead to wasted resources and time while deteriorating confidence in decision-making.

    Data observability, along with data governance, helps organizations manage data security and quality problems in a streamlined manner.

    Data Observability vs. Data Quality

    Data quality measures the completeness and accuracy of data sets to determine if they can be used in analytics and operational applications. Data observability, on the other hand, allows organizations to detect and fix issues in the data pipeline efficiently and quickly.

    For effective data management, an enterprise must take both into account. Earlier manual checking and cleansing of data sets helped determine data quality. But today, many of these tasks have been automated, thanks to modern data stacks.

    As a result, there’s been a shift in focus from data quality to data observability. With data observability, businesses can effectively monitor and troubleshoot their data pipeline. Without it, they are at the risk of relying on incomplete or inaccurate data to make decisions. This can lead to expensive mistakes.

    Finding the Right Data Observability Tools

    Undoubtedly, data observability is a critical feature of any enterprise that uses data. However, not all data observability tools are equally beneficial to your business. Look out for the following traits when choosing a data observability platform:
    ● Compatible: The tool must be compatible with your data lakes, databases and cloud storage solutions.

    ● Autonomous: Autonomous technology responds to stimuli without any human intervention. This is essential in a data observability tool as it helps in the early detection of anomalies and an instant response to alerts.

    ● Timely: A data observability tool must help you identify errors as early as possible. The best platform constantly monitors data health from the moment the data is added to the ecosystem to the end of its lifecycle. Early detection of errors stops issues before they become critical.

    ● Sophisticated: The best platforms leverage machine learning (ML) and artificial intelligence (AI) to identify hard-to-find issues.

    ● Other features: The tool must be able to collect, sample, review and process telemetry data across several data sources. It should serve as a centralized data repository, offer comprehensive data monitoring services and provide data visualization.

    Eventually, the right data observability platform depends on your organization’s observability engineering needs and unique IT architecture.

    Top Data Observability Tools

    Data observability platforms are still a budding product category. The good news is that over a half-dozen data observability companies now offer commercial tools with excellent features. These include the following:

    Observability Vendors, Overview,  Pros and Cons

    Monte Carlo Data is headquartered in San Francisco. Monte Carlo is the creator of the industry’s first end-to-end data observability tool.

    ● Provides comprehensive data observability capabilities.
    ● Offers a high level of features, including automated alerting, data catalogs, etc.
    ● Supports fully automated setup.

    ● High data volumes can lead to user interface issues.
    ● Huge amounts of variables restricted by different constraints can lead to computational inefficiency.

    Bigeye is an industry-leading data observability platform that allows teams to improve, measure, and communicate quality data clearly and quickly at any scale.

    ● An easy-to-use interface that facilitates data configuration while ensuring consistency and accuracy.
    ● Provides powerful API integration capabilities.
    ● Features a versatile dashboard with real-time tracking and monitoring of data quality metrics by multiple people.

    ● It can be expensive for smaller organizations.
    ● The tool might crash at times, requiring performance improvements.

    Acceldata offers tools for Hadoop, cloud services, and companies which include end-to-end data reliability, data pipeline monitoring, and multi-layer data observability.

    ● Provides fully automated data reliability checks.
    ● Offers a drag-and-drop interface to analyze data pipelines across several layers and platforms.

    ● Initial setup can be complex and difficult.
    ● Node addition and node removal require human intervention.

    Databand is an IBM company that offers proactive capabilities to identify and resolve data issues in the early stages of the development cycle.

    ● Provides cross-stack visibility. This means you can get an overview of all the data tasks from start to end.
    ● Offers standardized DataOps and end-to-end data lineage, ensuring data reliability and accuracy.

    ● The program requires a significant amount of space, which makes it difficult to install it on the user’s system.
    ● It requires constant software updates making it much heavier.

    Datafold has a unique ability to proactively detect, investigate, and prioritize data quality errors before they affect production.

    ● Allows you to turn SQL queries into smart alerts, keeping you updated on any issues that may arise.
    ● Automates regression testing by integrating the CI process through GitLab and GitHub.

    ● Limited integration options.
    ● Does not provide any support for data analysis and data science.

    Closing Thoughts

    Data observability is the backbone of data engineers’ ability to be agile with their products. If you want to modernize your data management practices and improve your data quality, data observability is the way forward. Without it, your data team cannot rely on its tools and infrastructure, as issues can’t be detected efficiently and quickly.

  • 5 Reasons to Move Beyond SRE to Observability

    5 Reasons to Move Beyond SRE to Observability

    Recently, I was asked on a podcast to explain the relationship between site reliability engineering (SRE) and observability and why I believe organizations need to move their focus from the former to the latter. Those are great questions, but to answer them accurately, I first need to point out that many organizations claim to be doing SRE or claim to be doing observability but aren’t.

    One big thing that SRE and observability have in common is that they are both often erroneously equated with monitoring.

    SRE ≠ Monitoring

    Google is credited with pioneering the SRE movement with the publication of the “Site Reliability Engineering” handbook in 2016. As the SRE movement gained momentum, organizations of all sizes pored over the handbook and began applying its principles. However, very few organizations are of the size and sophistication of Google regarding the scale of their IT operations and engineering resources.

    So, as these “normal” organizations realized how difficult it was to follow the Google SRE approach in its entirety, they often opted instead to simply apply what they could. For many, the chapter on monitoring became the focus, so much so that monitoring has become synonymous with SRE in far too many organizations today.

    Let me be clear: The role of a site reliability engineer is not to monitor alerts. The role of an SRE is to define how the engineering team should take ownership of their service. SREs are responsible for establishing a culture and creating engrained processes that are focused on the quality and reliability of infrastructure.

    SREs guide the engineering team on how an application or service should be observed and monitored, including what parameters should be looked at, what alerts they should have, and what routing path the alerts should go through. An SRE instructs on best practices, explores the tools that are being used and determines how to gather reliable data with consistency throughout the organization.

    Many organizations we work with say they want to do SRE this way, but they’re not there yet. They are still stuck on monitoring every single metric they can find.

    Observability ≠ Monitoring

    Similarly, a lot of companies I meet say that they do observability, but what they are doing is just monitoring. They monitor every single CPU of every node of every pod of every machine that is running. They have alerts for some of these, and they may even have a playbook for some of them. This is not how SRE is supposed to work, and it’s certainly not what observability is all about.

    More importantly, it’s not scalable as an organization grows to hundreds or thousands of developers and different teams that all share the same IT environment.

    Two Viewpoints: Horizontal and Vertical

    Now that we’ve established that SRE and observability are alike in that neither should be equated with monitoring, let’s contrast the two. SRE and observability are different in that they have different perspectives on the IT domain.

    SREs see the world horizontally. They conceive of their IT domain on a horizontal plane where they can observe the capacity of resources being used. They see the Kubernetes cluster, and they see the cloud utilization. And they care about ensuring that what is deployed into the production environment is secure.

    Although they see the pieces they manage on the horizontal infrastructure plane, they don’t see the ultimate impact on the application or customer on the planes “above.”

    Observability is looking at the world vertically—that is, seeing how an application is or isn’t meeting customer needs. The whole goal of observability is to know if my application is serving my customers at the service level that I agreed to. With a vertical view from application up to customer needs and business impact, I don’t care how many pods are being run, how many nodes are being deployed or what comprises the cloud infrastructure underneath my application, as long as it works.

    I do care about security insofar as I want to avoid introducing security vulnerabilities into the organization. But beyond that, all I need to know is that the application can meet the customer’s expectations and the service level objectives (SLOs) that we have established are being met, as well.

    Therein lies the heart of the matter: For a company to make the transition to observability, it needs to build the observability system around establishing and meeting SLOs.

    The Challenges of Transitioning to a Vertical Perspective

    Companies must move beyond an SRE and DevOps organization to being an observability and SLO organization. Only by doing this will organizations be capable of understanding the trade-offs within their IT environments between cost, security and availability and the ultimate impact those trade-offs will have on business objectives.

    Today, making this transition — moving beyond SRE to observability — is difficult for a wide variety of reasons, not the least of which are:

    Alert Fatigue. Companies today are monitoring too many things they shouldn’t care about, and as a result, engineers are suffering from alert fatigue. With this, you face the danger of a higher probability of missing the alerts that you do care about and making sure that you address them promptly.

    Ideally, observability solutions eliminate the irrelevant and unimportant alerts, add context to alerts that might need investigating, and correlate response data and docs to accelerate and coordinate the appropriate response of on-call responders.

    Tool sprawl. Many of the companies we work with use a wide variety of separate tools for observability. Observability requires access to the data all of these tools produce, and problems can persist if the tools operate in data silos.

    However, the problem can be solved if there is unified data collection across the toolset, ideally using OpenTelemetry. Cross-tool and cross-stack insights are particularly important for security purposes. Another issue with tool sprawl is that true observability requires that these tools, which each typically address just one aspect of observability, somehow all work together.

    Unfortunately, current systems and processes lack the “connective tissue” to offer a consolidated observability solution.

    Complexity. As we’ve seen in the results of our annual DevOps Pulse survey, the mean-time-to-resolution (MTTR) is going up instead of down, despite the number of tools that are available and the advancements made by the industry in recent years.

    Complexity is a large part of the problem, specifically concerning Kubernetes. In our 2023 DevOps Pulse Report, we found that almost 50% of respondents cited Kubernetes as their main challenge to gaining full observability into their environment.

    The adoption of Kubernetes is happening at lightning speed—even faster and higher than the adoption of cloud technology. But it’s not just the adoption of one Kubernetes—there are so many different flavors of Kubernetes, whether it’s Kubernetes running on managed services like AWS EKS or AKS or Kubernetes running on serverless with AWS Fargate. And that’s just to name a few of many, many types of Kubernetes out there.

    All of these options create flexibility, but wherever you create flexibility, you also create complexity. Complexity in the infrastructure increases the time it takes to troubleshoot and resolve issues.

    Cost. Most of us have figured out that products that include every feature under the sun are a waste of money. We don’t need expensive and extraneous clutter. Likewise, gathering and processing too much irrelevant monitoring data adds significantly to costs. Investing in observability platforms that are overloaded with bells and whistles you simply don’t need is similarly costly.

    The ideal observability platform taps AI and automation to help you easily see what’s most important to the business and respond quickly to core issues with precision and advanced insights.

    Culture change. Going from monitoring to observability is akin to making a cultural quantum leap. Observability requires a different organizational structure and assigns different responsibilities to the teams. The transition requires change management skills and patient, thoughtful execution.

    Though these challenges are significant, the observability industry is making rapid progress and organizations can expect to see solutions and technology providers that overcome these challenges shortly. So, be on the lookout for great opportunities coming down the pike.

    In the meantime, I challenge you to change your perspective from horizontal to vertical. Set your organization on the path to move beyond DevOps and SRE to true observability, where your teams will not just be alerting and chasing issues for troubleshooting but working to establish real SLOs and driving toward those to enable the strategic goals of the business.

  • Five Great DevOps Job Opportunities

    Five Great DevOps Job Opportunities

    DevOps.com is now providing a weekly DevOps jobs report through which opportunities for DevOps professionals will be highlighted to better serve our audience.

    Our goal in these challenging economic times is to make it easier for DevOps professionals to advance their careers.

    Of course, the pool of available DevOps talent is still relatively constrained, so when one DevOps professional takes on a new role it tends to create an opportunity for others.

    The five job postings shared this week are selected based on the company looking to hire, the vertical industry segment and, naturally, the pay scale being offered.

    We’re also committed to providing additional insights into the state of the DevOps job market. In the meantime, for your consideration:

    Indeed.com

    Sony Electronics
    New York, New York
    DevOps Engineer
    $125,000 to $135,000

    SimplyHired.com

    Fiddler AI
    Palo Alto, California
    DevOps Engineer
    $120,000 to $152,000

    CareerBuilder.com

    Leidos
    DevOps Engineer
    Olivette, Missouri
    $120,000 to $125,000

    LinkedIn

    Capital Group
    Irvine, California
    DevOps Engineer
    $92,423 to $147,877

    Dice.com

    CA-One Tech Cloud
    Austin, Texas
    DevOps Engineer
    $80,000 to $100,000