devopsy.kinsta.cloud

Tag: SLA

Increasing Use of SLOs to Enable Observability
Observability is a growing discipline among most IT and operations departments. To release stable software faster, operators need continuous visibility into metrics like performance, uptime and availability. As a result, engineers are increasing their use of service-level objectives (SLOs) across the board—a recent study found that 82% of companies are increasing their use of SLOs.

SLOs enable deep visibility into specific applications’ performance and are often used by site reliability engineers (SREs) to ensure quality and avoid service interruptions. Not only that, but various teams are correlating SLOs to aspects of the business to reduce cost and help direct decision-making. But while many environments have visibility, there are still gaps present.

Nobl9 recently released a global survey of IT professionals and executives that tracked the state of service-level objectives in 2022. Below, we’ll look at the key findings from the report and consider what they mean for the state of SLOs and observability in general.

State of Observability and SRE

In general, site reliability engineering is still maturing across organizations. Though only 31% of companies have adopted SRE, it’s slated for much future growth, as 46% said they plan to embrace SRE in the future.

These operators are now faced with many cloud-native observability tools, which are producing a sea of data in the form of metrics, logs and traces. A full 39% of companies use anywhere from six to 10 observability and monitoring tools and 35% use more than 11.

With so many tools now deployed, who is using this observability and monitoring data? Namely, at 74% of companies, observability data support operational requirements. Operations teams are the ones most likely to use SLOs to monitor uptime, performance and overall efficiency. After operations teams, security teams also use this observability data (71%), which makes sense as SLOs can inform incident response. Other areas that follow are customer support, compliance and capacity planning.

One interesting finding is that only 42% of companies use SLOs for service-level agreement (SLA) adherence. This indicates that SLOs are most often applied to internal optimizations and decision-making.

Hybrid Environments Complicate Visibility

Companies are mostly tracking SLOs to increase their visibility into networks (83%), databases (76%) and applications (75%). Other top areas include private cloud environments and legacy computing arrangements. But although observability is trending, organizations still lack complete visibility across the entire stack.

Of note is that nearly half (46%) of respondents said their monitoring and observability tools don’t provide full visibility into all of their company’s IT assets. For example, only 45% of companies have visibility into their containers and just 35% have visibility into their microservices architecture.

A lack of full-stack visibility may be complicated due to rising hybrid and multi-cloud conditions, as 78% say a hybrid cloud environment makes monitoring infrastructure more difficult.

Benefits of Tracking SLOs

The report defined SLO as “a performance and availability target set for a given system, application [or] service over time.” And the benefits are readily apparent for organizations following such targets—they can help increase performances, direct decision-making and help avoid outages.

As a result, 70% of companies are currently using SLOs in some fashion. Here are some benefits of tracking SLOs:
- Increase microservice performance: 87% said using SLOs for microservices architecture would improve service performance.
- Enable full-stack observability: 58% said some of their company SLOs are mapped to business operations.
- Improve business decision-making: 91% agreed using SLOs can help drive better business decisions.
- Prevent service interruption: 67% said their company prevented business interruptions thanks to SLO thresholds alerts.
- Reduce expenses: 90% indicated SLOs saved their company money.
Final Thoughts

More teams are tracking service-level objectives than ever before. And most companies (71%) that aren’t using SLOs now plan on adopting them soon. The data demonstrated that the observability market is an evolving area with room to grow. And, the same can be said about the SRE role.

Yet, it’s good to point out that not all companies will adopt precisely the same roles or monitoring procedures. Therefore, there will likely continue to be variance in who is using SLOs and how they are applied. Regardless, the study suggested that investigating SLOs has the potential to benefit the software life cycle in many ways.

The study, “Adoption of SLOs Grow to Increase Visibility and Drive Business Improvements” surveyed 309 participants around the globe in various sectors. The study was sponsored by Nobl9 and conducted by Dimensional Research. For more information, you can download the report here.
May 17, 2022
How to Improve Your Uptime Strategy
Outages happen, it’s inevitable. But, unplanned downtime often comes with substantial costs—not only in terms of recovery and revenue loss, but also customer happiness, brand reputation and employee morale.

While there is no foolproof way for companies to avoid outages, there are steps they can take to help improve uptime. Development teams should determine what measurements and thresholds for uptime are sufficient for the company, take preventative actions to reduce the mean time between failures (MTBF) and implement tools that provide awareness of coding errors to minimize the mean time to repair or resolve.

Defining and Measuring Uptime

Uptime is defined as the time when an application or service is operational. A service-level agreement (SLA) between service providers and customers defines the level of uptime providers have to meet. But “operational” can mean different things to different organizations. For example, a provider may have an SLA for how long it takes its system to process an event. Events processed too slowly violate the SLA even though the system is still operational at some level.

Companies that provide digital services and applications should clearly define SLAs with customers and then measure their performance against those objectives. Dropping below the agreed-upon expectations can lead to customer frustration and contract termination. Also, the time and effort development teams spend updating customers, tickets and status pages can distract them from working on new service features.

In SLAs, uptime is measured by “nines,” with each nine hard to achieve as it means significantly less downtime. Here’s a common breakdown for measuring uptime:
- One nine (90% uptime) is 36.5 days of downtime per year.
- Two nines (99% uptime) is 3.65 days of downtime per year.
- Three nines (99.9% uptime) is 8.77 hours of downtime per year.
- Four nines (99.99% uptime) is 52.6 minutes of downtime per year.
- Five nines (99.999% uptime) is 5.26 minutes of downtime per year.
Not all companies need five nines. Because each additional nine requires increased effort and cost, development teams should determine how to balance this investment of time and resources in a way that best meets business needs, while focusing on resolving issues quickly when they occur.

Preventing Errors in Production

Validating code can prevent errors in production and minimize MTBF. This type of testing needs to be done in the early stages of the development cycle. To reach the three, four and five-nines levels of uptime, automated testing is necessary. While such testing involves an upfront investment in writing unit tests, it saves time and increases efficiency, as a failing unit test means a specific block of code is not working as expected.

After validating code, another step developers can take is to use a canary deployment. This is a strategy where new code is pushed out to a subset of users. If something goes wrong, then only a small percentage of users are affected, and developers can roll back to a stable version. Once developers are comfortable with the results of the new version, they can gradually deploy the new code to all users.

Uncovering Context with Errors

Receiving an alert for an error is not enough to enable quick remediation. Developers need to know when errors occur, what the impact was, and where the errors reside in the code. Having this level of awareness and context minimizes mean time to repair, and the best way to achieve this is through application monitoring.

Monitoring provides developers with automated real-time insights into how their service is performing. When alerted to issues, developers can use source code management information with monitoring insights to identify the change that caused the problem and get the issue to the appropriate developer to resolve it. Developers can’t rely on manual processes and customer feedback to be alerted to issues. These methods are not fast enough. It can take hours to receive and understand an issue and then manually roll back to an earlier version or forward to a new version.

Code can be complex and fragile. It touches virtually every aspect of business and life. When code stops working, it’s expensive and disruptive. Downtime, even for a brief moment, has a disastrous impact on reputation, retention and revenue.

To improve uptime, developer teams should adopt code validation and monitoring strategies. By balancing prevention and awareness of errors, they can reduce broken code in production, decrease time to resolution when errors do occur and improve service uptime.

— Neil Manvar
April 6, 2020
Optimizing AWS Performance Over the Long Term
Ask most folks how to optimize performance on AWS, and they’ll talk about things such as autoscaling and rightsizing.

Those tools are certainly part of an effective AWS performance optimization strategy. On their own, however, they are not enough to guarantee performance and reliability over the long haul.

That’s because tools and processes such as autoscaling and rightsizing only help you optimize the performance of applications that are running currently, or are about to be deployed. They do little to ensure that performance remains optimal—especially as cloud services, cost structures and architectures evolve.

With that challenge in mind, let’s discuss how to get the most out of AWS performance over the long term.

What is AWS Performance?

Before delving into long-term AWS performance strategies, let’s make clear what we mean by performance. A high-performing application is one that:
- Meets or exceeds user expectations regarding responsiveness and speed.
- Meets or exceeds SLA requirements for availability.
- Is able to scale seamlessly as performance requirements change.
- Does all of the above in a cost-efficient manner.
These aspects of a high-performing AWS application are important to emphasize because they reflect a holistic approach to performance. Sometimes performance is defined narrowly, in terms of application responsiveness and/or availability. Those are part of the performance equation, but in order to maximize performance over the long term, you need to think about other dimensions of performance, like the role played by scalability and cloud spend.

Long-Term AWS Performance Considerations

To optimize AWS performance over the long term, you must take a variety of factors into consideration.

Handling AWS Service Changes and Upgrades

AWS is constantly evolving. A performance strategy that works well today may not be as effective tomorrow if AWS introduces a new service or changes the features of an existing service.

The challenge here lies not just in updating your app and your cloud configuration as AWS changes how a particular service works. It also requires constantly evaluating whether the service you are using is the best fit for your application.

For example, think back about five years to when AWS Lambda (AWS’s serverless computing service) appeared. At the time, if you were running your apps inside virtual servers, you might have been able to improve overall performance by moving some of the workloads to serverless functions—not because anything in your virtual servers had become less efficient, but because better efficiency opportunities opened up with the addition of a new AWS service.

Thus, being constantly aware of the various services in AWS and the opportunities they offer is key to optimizing long-term performance.

Plan for Changing Cost Structures

AWS not only changes its menu of service offerings frequently, it also updates pricing. Since cost optimization is one key component of performance optimization, you must ensure that your cloud workloads are running in the most cost-efficient way even as cost structures change.

Staying aware of new cloud services (such as lower-cost S3 storage tiers) when they appear is one way to do this. But you also need visibility into the tremendously complex world of AWS pricing in all of its region-specific details.

Plan Long-Term for OSS Dependencies

Open-source software is something else that is constantly changing. If you rely on open source code to help run your AWS applications, you need to stay on top of changes in the projects you depend on.

If you are using the same version of an open-source application or module that you built into your application when you first deployed it, you may be missing out on performance optimization opportunities that have arisen since then.

Here again, constant awareness of and visibility into the complex open-source ecosystem is essential for ensuring that you are not missing out on optimization opportunities.

Application-Specific Performance Monitoring

Every application is a special snowflake and you need to keep that in mind when it comes to the way you track and optimize application performance.

The metrics or KPIs that work for optimizing the performance of one app may not be ideal for another. The pre-deployment testing routines you use for applications might vary, too.

So, make sure to tailor your monitoring strategy to your applications. Doing so is the only way to ensure you have the greatest, most relevant level of visibility into application performance over the long term.

Backup and Disaster Recovery

No matter how well your AWS workloads perform, sooner or later something will go wrong and they will fail. Whether the failure leads to a critical availability disruption or a mere hiccup hinges in large part on whether you have a backup and disaster recovery plan in place beforehand.

For AWS apps, an effective backup and recovery plan entails more than just backing up data to storage buckets and downloading it to rebuild servers. It should also take advantage of opportunities like the ability to create image-level backups of virtual servers and restore them automatically to EC2 if disaster strikes.

And of course, as AWS changes, you’ll need to keep your backup and recovery strategy in sync.

Changing SLA Requirements

Last but not least, keep your SLAs in mind. The SLAs you have to support today could change in the future—and by extension, so will the levels of availability and responsiveness that your users expect.

For this reason, make sure your AWS performance strategy is capable not only of meeting the SLAs you currently have in place, but also adapting and scaling as SLA requirements become more rigid.

Conclusion

Although we tend to think of performance within the context of the present, it’s really a long-tail game. Sure, you can autoscale your workloads or use rightsizing tools to help choose the best type of EC2 instance for right now. But truly maximizing performance requires you to think longer-term and be prepared to react with agility as new performance challenges and opportunities arise.

This sponsored article was written on behalf of Eplexity.

— Chris Tozzi
November 26, 2019
AIOps: The Path to Greater Resilience and Uptime

Large organizations with complex IT infrastructures are facing a significant challenge when it comes to data. The complexity is such that many IT operations teams are struggling to detect, or predict, the complex failure conditions and circumstances that can adversely impact resilience and uptime.

Monitoring a multitude of systems is a complex task, as the data generated by any system can be vast and far too overwhelming for manual IT operations processes, particularly in relation to analyzing, correlating and prioritizing alerts and remediation.

Companies have much to gain from a better understanding of the data being created from monitoring tools and practices: increasing the resilience of systems, and the speed of remediation cycles delivers significant cost and time savings. Harnessing advanced technologies to deliver more intelligence to teams in charge of performance monitoring is becoming increasingly important, which is where AIOps can deliver significant value on top of existing monitoring and automation tools.

Cutting Through the Noise

Telemetry tools deliver staggering amounts of data. The Microsoft Azure telemetry platform, for example, records 10 petabytes of data per day. When dealing with this volume, the ability to intelligently decipher the severity of alerts becomes vital, especially as many of them may be duplicates or not meaningful in any way. Yet it is IT operations teams that are tasked with understanding how systems are performing, and where resources should be diverted to ensure services are always up and running. To do this effectively, the root causes of alerts must be identified quickly and correctly.

AIOps is increasingly becoming the go-to solution to the problems inherent in dealing with big data. By applying data collection, data modeling and data analytics techniques and using machine learning algorithms to establish patterns, the noise of the data is reduced as operations teams gain more intelligent insights into alerts. So, providing more context to issues flagged within systems is the goal. Eventually, somebody has to take action when an alert is created, so the more intelligence engineers have, the easier it will be to remediate.

The less context delivered with alerts, the more time engineers will need to resolve the issue, as the manual remediation process will likely involve interaction with other teams. Take application performance monitoring as an example. In my experience, operations teams do not have an end-to-end understanding of the applications that are live in the production environment. If an alert is created, the ops team will typically seek assistance from a member of the developer team who built it, or another colleague with a deeper understanding of the application, which of course extends the lead time of the remediation process.

Key Considerations for Deploying AIOps

By establishing data patterns through machine learning and other advanced analytics technologies, the following key drivers for alerts assessment using AIOps can be established:

Frequency: Discovering how regularly alerts are created and whether it warrants automation for analysis, correlation, prioritization and remediation.

Coverage: Telemetry is of course vital to AIOps, as the machine learning algorithms will only be as good as the data they ingest. Putting in place the right telemetry tools is therefore vital to ensure the right data is being gathered. Failing to do so increases the risk of missing critical issues.

Impact: Not all problems require equal energy and resources. Certain issues might only occur every few months and may not be critical. In such situations, it is not prudent to invest significant amounts of time and money into developing the machine learning algorithms that will identify where and when an alert will be created. Again, cutting through the noise of the telemetry will help to identify high-priority alerts.

Probability: What is the likelihood of certain issues recurring? Can automation tools, informed by the machine learning algorithms, be implemented to deal with these, i.e. learn where and when they are likely to occur so they can be dealt with by the AIOps platform autonomously?

Take, for example, the issue of CPU or memory going beyond a certain threshold. Rather than manually contacting customers when an issue is flagged to inform them that services are about to go down before being spun up again, an AIOps platform can automate this process, so remediation happens seamlessly.

Latency: Automation of the remediation process for an alert deemed critical requires increased investment in AIOps. Getting good data from the early stages of AIOps processes will reduce the amount of time taken to both flag and deal with a problem. Containerized microservices, for example, take very little time to recover, so very little investment is needed when applying AIOps tools that automate remediation. Database recovery, however, is a more complex process, the automation of which would require significant investment.

What’s the Customer Value?

For service providers, AIOps is essential if they are to attain the highest levels of resilience, the measure of which all comes down to the Service Level Agreement (SLA). The SLA is measured by a percentage that corresponds to the time systems and are up and running over a year. So, 99.99% uptime is good, 99.999% is great, but 99.9999% is where service providers want to be. To put this into context, downtime for an entire year in the latter scenario amounts to just 32 seconds.

Increased automation is the only way to achieve optimal SLA. To do this, service providers are increasingly turning to AIOps solutions.

Operations teams are in charge of the IT strategies that align with business objectives. For service providers, a key part of this is, of course, ensuring maximum availability and optimal performance of the customer-facing environment. Understanding when and where faults appear, and improving resilience against them, is the best way to ensure this. That is where AIOps truly adds value.

— Prashant Jain

November 4, 2019
Service Level Agreement

Here are some other ROELBOB’s you might like

A Unit Testing Dillema

The Pink Slip

Another Pink Slip

Hot off the Press – Other great devopsy.kinsta.cloud content

Successful technology leaders will embrace the citizen developer

Your DevOps subculture

Upcoming devopsy.kinsta.cloud Events

DevOps Connect: cdSummit SoCal

Security automation for DevOps

— ROELBOB

November 20, 2015
DevOps culture meets the SLA

The goal of every SaaS provider should be 100% availability – it should be ingrained in their culture. But it’s far more common to see a series of nines. Numerically, these are clearly not the same value and culturally, they are a world apart. When you have an availability target that is less that 100%, you are explicitly tolerating some amount of downtime. It changes the requirements, the perspective of all those involved in delivering the application to your customers. When we think of the nines, I think we invariably fall into thinking about the differences between them – especially the technical differences. But to get to 100%, you need culture. A culture that is fanatically dedicated to achieving that goal.

A review

This is tired ground, but let’s take another look at what the most common SLA measurement is and what it means. System availability is measured as a ratio of the number of minutes of actual availability over some period of time – a week, a month, a quarter, a year. We usually talk in terms of the monthly SLA for a 30 day month. 99.9% availability is 43.8 minutes of downtime per month. 99.99% is 4.32 minutes per month, and 99.999% is just 25.9 seconds per month.

What I find interesting about these numbers is the implication of trying to assure a higher SLA. You can achieve 99.9% with process and dedication. But that won’t get you to 99.99%. To achieve that, you need at least fail-over automation. To get the fifth nine, you need self-healing automation. Each of these numbers represents an order of magnitude increase in availability. Likewise, the effort and expertise required to attain that availability is greater.

What’s your relationship to the SLA?

When there is a disruption of service, the first obligation an organization has is to restore that service as quickly as possible with the least amount of risk. In mature organizations, there are typically automated and self-healing processes that provide a first line of defense for the most common types of failures. But there are always unanticipated problems that can manifest at any time. The expertise required to address those varies depending on the nature of the problem. It might require someone who primarily identifies as development, or dba or security. Everyone else attending to that problem is secondary to the key resource required to fix it. I have spent many hours feeling helpless during outage war rooms because I was not person with the specific skills required to solve the problem. When you expand this problem out, you begin to see that in order to prevent the outage from occurring again, you need that key resource to be thinking about outage/problem mitigation all the time, rather than in response to a specific incident.

Sometime, during times of trouble like this, I’ll be approached and asked what we should do to address this problem. On many occasions, my answer has been, “we need to build a system that doesn’t have outages.” After an awkward silence, the asker begins to realize that the askee is actually quite serious. And I am. The problem is that to truly attain that goal, or something reasonably close to it, you need everyone involved at all times at some level.

SLAs are meaningless

About 15 years ago, I experienced a catastrophic failure in a name brand storage array I inherited by way of employment. It was one of those managed, completely redundant, fault-tolerant arrays that consumed a considerable amount of OpEx with a 99.5% availability in the SLA. When it inevitably failed (you knew that was coming, right?) and I looked up the contract, I found that we could not claim a material breach of performance unless the availability of the storage device fell below 85%. I was stunned. My business would be dead long before 85%.

Likewise, you can have a 99.95% month, but if the tiny amount of downtime happens to impact your biggest and most demanding customer, the SLA is immaterial. They become angry and are at risk as a customer because you have failed them. What I learned as an operations executive is that the commitment behind the SLA is far more important than the SLA itself. Put another way, you can make bad situations into positive ones if you handle them exceptionally well every time. That means ruthless dedication to uptime from everyone in the company, from rank-and-file individual contributors to the CEO, all the time, not just during the incident. If that really is a goal – and it’s infused into the culture of the company – everyone you interact with will see it and understand that commitment.

There’s another artifact of SLAs that can be detrimental. When the focus is on the availability number, the target is too low. The goal of every SaaS provider should be 100% uptime. When your goal is 99.9%, you effectively have a budget of downtime that you can draw from while still exceeding your goal. Here’s an example from my own experience. I once had a service that required periodic downtime for maintenance. One month, I had incurred 30 minutes of downtime via an unplanned incident against that service – a service with a 99.9% SLA. We also had a software update planned for that month that was going to require 20 minutes of downtime. I had the update moved to the following month so that it wouldn’t blow the SLA for the current month. It didn’t make much sense at the time and didn’t feel very customer-centric either, but as Dr. Eliyahu M. Goldratt wrote in The Goal, “Tell me how you will measure me and I will tell you how I will behave.” I had a chance to hit the SLA for that month, so I did everything I could to hit it. Given that I deferred a software update that would have delivered new features to my customers, the choice I made seemed arbitrary and was counter to being a customer-focused organization.

It takes a culture

One truly exciting thing about the DevOps movement is that it advocates the spread of knowledge throughout the organization. From my selfish operations perspective, that means that individuals who formerly would only be contacted in the event of dire emergency now have a much more visceral connection to the availability and performance of their services. By being exposed to the operations experience, those formerly external parties now see challenges and problems. And most people in our industry love a good challenge. What I get from this exchange is the injection of new perspectives and a diverse and healthy debate about subjects that have long since been worn out living only in the realm of operations. To have the attention of that level and breadth of expertise is a fantastic opportunity for me to evangelize availability and performance. More importantly, if all that talent is super motivated to achieve 100% uptime, performance and availability are going to follow. Anything less is a compromise and I don’t want to compromise on behalf of my customers. I want them delighted 100% of the time.

May 6, 2014