Tag: devops teams

  • Why MTTR is a Vital Metric for DevOps Teams

    Why MTTR is a Vital Metric for DevOps Teams

    In DevOps, every second counts. System failures mean your engineers lose valuable time improving software and developing new features. So how do you effectively detect and manage these issues? Mean-time-to-resolve (MTTR) can provide insight into how effectively your DevOps team responds to incidents and how reliable your software is. It’s difficult to know how your team can improve or determine a performance baseline if you don’t have data to analyze your incident management workflows. In this article, you will learn about this valuable metric, interpret it and leverage this information to minimize downtime.

    Why MTTR Matters

    Whether you’re a legacy organization striving to modernize, a startup looking to gain an edge or somewhere in between, MTTR is a vital metric for your organization. It can indicate whether your processes are standardized and can highlight crucial areas for optimization. Most importantly, mean-time-to-resolve provides insight into how stable your software is and how responsive your team is over time.

    Mean-Time-to-Resolve and How to Interpret it

    MTTR can stand for many different things: Mean-time-to-repair, mean-time-to-recovery, mean-time-to-restore or mean-time-to-resolution are just a few interpretations of this versatile acronym. Each of these metrics examines a different aspect of addressing an incident. For example, mean-time-to-repair is the average time it takes engineers to fix a problem, calculated from the moment they begin to the moment a change is pushed to production. But this metric does not account for the amount of time between when an alert comes in and when work begins.

    Mean-time-to-resolution is the most high-level of these metrics—it considers the average time from when an alert comes in until the incident has been resolved, including postmortems and implementing process changes to avoid the same issue in the future. While the standards for this metric can vary by industry, the highest-performing DevOps teams see an MTTR of less than one day.

    Because it is such a comprehensive metric, a high mean-time-to-resolve measurement might indicate problems with alerting or that your engineers are spending a lot of time on repairs. Therefore, it’s essential to look at MTTR over time and analyze each component of your incident management workflow: Time to alert engineers, diagnose the issue, test fixes, ship to production, conduct reviews and learn from the incident.

    It may also be helpful to examine mean-time-to-resolve in conjunction with other metrics. To determine if your DevOps team is facing production challenges, evaluate your change failure rate (CFR) to see how many releases result in a downgraded service. Other DORA metrics, like deployment frequency and lead-time-to-changes, are perfect companions for mean-time-to-resolve.

    To establish the reliability of your software, you can look at mean-time-to-resolve beside mean-time-between-failures (MTBF), which calculates the average amount of time between incidents. If you’re updating your software often, compare mean-time-to-resolve with mean-time-to-failure (MTTF), which measures its duration before a program needs to be redesigned for functionality. To gain insight into your alerting processes, examine mean-time-to-detect (MTTD), which evaluates the time it takes your team to recognize that an issue exists.

    How to Improve Mean-Time-to-Resolve

    Alerting is the first stage of responding to an incident and should be one of the first areas to target when working to reduce mean-time-to-resolve. Ensure alerts are actionable and that DevOps team members have the tools they need to respond immediately. A straightforward escalation process is essential: Define responsibilities for each member and train the team in one another’s roles so that the process never grinds to a halt if someone is unavailable.

    Preemptive monitoring can help you get ahead of problems before they arise—by proactively checking for potential incidents, you can avoid unexpected downtime.

    The best way to improve MTTR is to standardize your operating procedures with runbooks. Without runbooks, DevOps teams have to respond without a clear direction and spend time messaging one another for information—they can’t act immediately. With runbooks, however, your organization’s knowledge base is centralized and accessible to all team members, enabling them to respond as soon as an issue comes up.

    If you’re already using runbooks, consider automating responses. Automation not only improves your mean-time-to-resolve but will give your DevOps team more time to devote to implementing long-term changes that improve the stability of your service.

    When implemented correctly, mean-time-to-resolve is a proven metric that can indicate how effective your DevOps team responds to challenges and how reliable your software is. But be careful not to incentivize the wrong behavior in a quest to minimize this metric. You could, for example, quickly reduce MTTR by eliminating all alerting protocols, but that would also mean poor service for your end users and likely result in poor team morale. Examine this metric over time and aim for slow and steady growth rather than a quick fix.

  • Positioning ML Devs and Teams for Success

    Positioning ML Devs and Teams for Success

    Intelligent applications are (by their very nature) complex. While conventional software basically consists of one thing (code), intelligent software involves code, models and data. As previously discussed, three distinct fields—DevOps, MLOps and DataOps—have evolved to govern each of these interconnected disciplines. 

    Moving through the ML life cycle quickly and efficiently requires collaboration between teams in each of the above disciplines. Organizations should strive to make this collaboration as frictionless as possible to position their ML efforts for success. What follows are four places where we believe organizations should focus: 

    • Infrastructure
    • Team structure
    • Tools 
    • Project ownership and KPIs

    Build Dedicated Infrastructure for Each Pillar

    Infrastructure software is not one-size-fits-all across the three pillars of code, model and data. For example, DevOps teams have historically played the infrastructure role in software development, supporting application or product services. Think of regular software engineering teams who are supported by DevOps, producing different kinds of application services. App infrastructure is so baked-in to software development that most of us take it for granted.

    Just as regular software services require DevOps, data and model services require corresponding infrastructure. Data infrastructure is typically concerned with managing tools (such as Airflow installations, Spark clusters and feature stores), building data pipelines and ensuring that data services remain smooth and operational for data consumers. 

    On the other hand, model services are chiefly concerned with creating, running and organizing models in such a way that others can use them efficiently. Since the AI/ML life cycle is incredibly complex, a host of factors need to be considered (e.g., hardware, parallelization libraries, model packaging formats, etc.) to help data scientists and engineers do their jobs efficiently. Robust ML infrastructure along with a dedicated ML infrastructure team goes a long way in helping organizations scale their AI.

    Optimize Team Structure

    Best practices for ML and AI are continuously evolving, and organizations have tried a host of team structures over the years. We have noticed two popular structures for intelligent application development teams emerged in recent years:

    • Pods

    Pod structures involve smaller teams—usually a data scientist, a data engineer and an application software engineer—working together closely to deliver an intelligent feature or product. Due to their tight-knit nature, these teams are positioned to iterate extremely quickly, which often leads to better products and features. However, the closely-knit nature of pods also comes with a downside, as knowledge sharing between pods with different objectives can be difficult.

     

    • Center of Excellence

    In a center of excellence (or COE) structure, all data scientists or data engineers are “pooled” and then assigned to different products or features depending on project requirements and circumstances. In contrast to pods, COE structures tend to share knowledge very easily. However, this arrangement can lead to slower iteration cycles because team members are not dedicated to particular products or features. 

    In our opinion, both pod and COE arrangements can be effective depending on the realities of an organization. Choosing an optimal team structure depends on how the rest of the organization is structured and how reporting relationships are set up. 

    Select Appropriate Tools

    To select the best possible tools for your organization, choose those that foster collaboration and uphold safety across code, model and data teams. 

    The right tools help teams work quickly and efficiently. Teams should be able to discover pertinent information for themselves without having to pull others off of whatever they are working on. For example, a data science team should be able to discover what data access or learn what features exist. Similarly, application teams should be able to discover what models exist, what versions are the latest or how and when to use models. The more that you can make your code, model and data self-serve, the faster teams will be able to build products that are ready to be put into production. 

    Furthermore, proper tools make debugging faster and easier. If you’ve been around ML-enabled features or products for any length of time, you’ve probably been bitten by undeclared dependencies and apps breaking due to upstream changes. In an intelligent app, finding the root of a problem can be extremely difficult because the root cause can exist in any of the three pillars or any combination of the three. Robust tools provide logs and information to help discover what went wrong more easily.

    Finally, governance should play a key role in tool selection, as safety should be a chief concern for ML teams. For example, consider a model that predicts the demographics of a user based on their app usage. While these predictions might be considered acceptable in certain applications, if a model is being used to make financial recommendations or approve credit applications, use of certain demographics may violate various consumer protection laws. Since a software engineer using this model might not be aware of the nuances related to its use, baking in safety checks on model and data use becomes vitally important. 

    Establish Ownership and Determine Appropriate KPIs

    Whether we are working at a company or in an academic lab, our goal always remains the same: build the best product. 

    But “best product” can mean a lot of different things depending on the use case. For example, sometimes teams may be willing to trade a bit of accuracy if a model fails in predictable ways (e.g., in health care applications.) By contrast, an e-commerce recommendation system can optimize for overall model accuracy even if a model isn’t quite as stable. As a result, choosing the appropriate KPIs for model optimization requires careful attention.  

    Similarly, choosing appropriate metrics that make sense for the product and business usually requires a high-level view of a number of factors — user expectations, revenues, user complaints, etc. In our experience, product owners are usually in the best position to make these decisions. KPIs for each of the pillars supporting the intelligent app should be based on key metrics determined by product owners. While these can range, we’ve seen the following KPIs work well:

    • Code — Number of releases, number of bug fixes, number of features that have shipped, mean time to resolution
    • Model — number of model iterations shipped, model quality in production, model performance SLAs 
    • Data — frequency of data refreshes (e.g., every day or every month?), number of data quality issues, speed of access to data

    The key point here is to drive all of these based on the product KPIs and to focus on the agile principle, which is how quickly are you moving and how many deploys are happening. 

    Know what metrics your team is looking to optimize so that you can invest your resources in the right spot. 

  • Protecting Data from Insider Threats

    Protecting Data from Insider Threats

    “Insider threat” can be defined as the threat an employee or a contractor will use their authorized access to, wittingly or unwittingly, do harm to an organization’s security. Certainly, as organizations become data-driven, embrace data democratization and release new applications at breakneck speed, the probability of an insider threat is rising sharply.

    A common example of an insider threat is granting permissions to on-call SRE engineers who may need to access system tables across multiple databases, inspect contents of S3 buckets and make changes to the production environment to fix availability issues. Every two weeks, the composition of the team may change, and the composition of services and databases may vary dramatically.

    While it is impossible to completely safeguard against insider threats, organizations can better protect their data by investing in best practices that (a) reduce the likelihood of such a threat playing out, (b) limit the blast radius from an insider attack and (c) help detect the attack as quickly as possible. The third point is especially important – the Ponemon Institute’s recent Cost of Insider Threats 2020 Global Report claims that it takes, on average, 77 days to detect and contain an insider attack.

    A DevOps Framework for Protecting Data From Insider Threats

    For security measures to be effective, they must be non-invasive, especially in agile organizations where DevOps teams are expected to push releases several times a day.

    Implementation of Access Policies-as-Code

    Implementing access policies-as-code automates workflows for user and service access authorization. Using this approach, manual policy practices are replaced with “code” (policy files readable by your team and by the systems that enforce the policies) so that the right people on your team can request, approve and publish policy changes instantly.

    As an example, here’s how the above scenario may be implemented as code:

    Engineers can add themselves to on-call status with a code push in GitHub to a Terraform script, with only the team lead authorized to approve this action. (See example 1, below.)

    insider threats
    Example 1

    Using frameworks like Sentinel, changes to production environments can be limited to only on-call engineers. (See example 2, below.)

    insider threats

    Example 2

    Similar policies can limit only those engineers who are able to read unencrypted content of S3 buckets by fetching keys from Vault. (See example 3, below.)

    insider threats

    Example 3

    The above examples demonstrate guardrails that allow engineers to perform their duties, while drastically reducing the administrative burden on leads, resulting in fewer errors. An additional benefit of this approach is that it forces a public review of all changes to policies, which makes the security model more robust.

    Zero Standing Permissions

    Zero Standing Permissions (ZSP) is a DevOps-friendly security tenet that requires any administrative privilege to be available to users only when needed, and only for as long as needed. It enables teams to configure access, via administrative privileges, to highly critical components of their infrastructure for a limited duration, with restrictions on the scope of those privileges.

    Here are a few examples of how teams can implement ZSP for their databases:

    • Create an SRE role on the database that can update and delete database tables and a private Slack channel of db-admins that can allow on-call engineers to assume this role.
    • Implement a service that can be used to request and grant access via this SRE role for a default of 60 minutes. After that time, the access is automatically revoked and needs to be granted again.
    • Further implement policies in BrewOPA that restrict an SRE from dropping tables and databases, which avoids any disasters.

    Audit Trail for Privileged Activity

    SIEMs are increasingly the nerve center of every security organization. While effective threat-hunting and incident response requires several years of expertise, DevOps teams can significantly bolster the security of their cloud platforms by investing in log collection and basic analytics for all privileged activity in their environment.

    These tools include services such as the following:

    • Session recording for all Linux boxes using Cmd
    • Logging activity across databases with Data Cloud services
    • Centralizing and collecting all environment logs using services like CloudTrail.

    These logs can be visualized in tools like Kibana, where teams can look at activity pattern anomalies. These could include:

    • Someone opening up an FTP connection to outside the organization
    • Creating a manual copy of tables outside of the database
    • An increased number of authentication failures across resources

    While the concept of “least privilege” as a safeguard against insider threats to data has existed for some time, the advent of cloud-native and DevOps methodology has exacerbated the challenge of implementation. In recent years, security-as-code and infrastructure-as-code have emerged as good models for implementing security solutions, and are rooted in the above framework. The future encourages collaboration across various stakeholders, documents security policies for easy review and builds on existing tools and workflows. It will help limit insider threats, limit privilege sprawl and prevent “user fatigue.”

  • 3 Ways to Support DevOps Teams in Remote Work

    3 Ways to Support DevOps Teams in Remote Work

    As we settle into remote work for the long haul, it’s crucial to consider how best to support DevOps teams. While DevOps employees face similar challenges working remotely as any other department, they also face increased pressure to deliver as IT professionals, supporting the entire organization through an ever-changing work environment.

    By nature, remote work makes it more difficult for your DevOps team to do their jobs. Team members are working in various environments, but also distributed across geographies. And in this new frontier, DevOps must provide service as usual, continuing their development projects and ensuring communication and alignment among departments when it comes to the organization’s technology and IT needs.

    Even more, in a distributed, remote workforce, disruptions to application performance and security issues are more likely to occur and DevOps professionals must be equipped to identify and correct them quickly. For these reasons, it’s critical that these teams have the right tools and support to remain operational, communicate effectively across departments and neutralize threats—keeping applications and the organization running.

    How to Support Remote DevOps Teams

    Remote success looks different for every DevOps team based on organization size, industry and business need. However, the following strategies can be applied universally and may even serve as building blocks on technological foundations you already have in place:

    • Continue migrating operations to the cloud. Your cloud infrastructure is the backbone of any remote support plans for DevOps teams. Migrating operations to the cloud is the biggest enabler in making sure your team is not tied to any physical location and can do their jobs without relying on a single data center or headquarters. For example, deploying your open source data infrastructure in public clouds not only makes it easier for team members to access the data but also increases developer and operational velocity, which accelerates application innovation. While most organizations are well into their migration to the cloud, it’s critical to accelerate the process to expand across more operational areas. Traditional work environments likely won’t return to fully in-office experiences. Rather, many organizations will implement a hybrid working model. Refining and expanding your cloud infrastructure will help you make that transition and continue supporting an in-office, remote or hybrid team in the future.
    • Automate security and performance monitoring. DevOps teams are under pressure to scale solutions for remote work, such as company VPNs for network access and secure application logins. They must also be alerted to and able to track any security issues that arose during the hasty transition to remote work. Automated monitoring of software performance and trends keeps teams ahead of threats and minimizes deployment issues. The software sends alerts and allows teams to view data in real-time. Even more, machine learning technology can analyze the data and identify trends about application performance, the cloud environment and data usage. This level of monitoring and analysis is only possible with automation and makes it much easier for DevOps teams to meet IT service requirements as the remote technology landscape rapidly changes.
    • Increase communication and collaboration efforts. It’s crucial that DevOps teams are in constant communication with key departments across the organization, including security and executive leadership, as well as with other employees to discuss what’s working and what needs to become a priority for change. To help support knowledge sharing across a remote team, first ensure you communicate all application metrics clearly so every member of the team understands and is aligned on projects and next steps. Dashboards can help in this arena, collecting and converting key data points into visuals. And while you can easily create and host meetings with conference calls and video conferencing software, informal face-to-face conversations are harder to re-create—and that’s where ideas are born. To combat this loss, arrange for non-structured discussions without strict agendas or topics to encourage idea sharing, creativity and inspiration among employees.

    Remote work has broken down the traditional DevOps team structure, but this new way of work provides an opportunity to build something better—more time for independent, focused work, a superior work-life balance and modern cloud-based solutions for daily tasks. Supporting DevOps teams as they grapple with the new pressures of remote work is the key to ensuring your organization runs smoothly during the transition—cloud infrastructure, automation and communication are just a few of the strategies to do so.