devopsy.kinsta.cloud

Tag: SREs

  • How SREs Benefit From Feature Flags

    How SREs Benefit From Feature Flags

    When you think of who uses feature flags, your mind most likely goes to software developers. In general, feature flags are closely associated with software engineering.

    But site reliability engineers (SREs), too, can benefit. SREs may not be the ones to create feature flags, but they should work closely with developers to ensure that the applications their teams support include feature flags.

    Here’s why feature flags benefit SREs, and what SREs can do to make sure their organizations take full advantage of them.

    What Is a Feature Flag?

    A feature flag—also sometimes called a feature toggle—is a technique in software engineering that allows developers to turn specific features on or off at application runtime. In other words, they let you control which features are available within your application when it’s deployed to production.

    Feature flags can be configured in such a way that they enable features to be toggled on or off for specific sets of users, rather than an entire set of end-users. This is useful if, for example, you want to turn on an experimental new feature for users who have agreed to test it without exposing that feature to your general user base.

    Usually, developers and SREs use feature flags to turn features on and off manually. But you could automate the process by, for example, writing a script that automatically shuts off a feature if your observability tools determine that the feature is causing a problem for users.

    Why Developers Love Them

    The main reason why feature flags are popular among developers is that they make it easy to implement and experiment with new features while keeping the development pipeline simple.

    Without them, developers would have to create a new application release every time they wanted to implement new features. If any of the features in that release turned out to cause a problem, they’d have to remove the features from the code or fix the buggy features, then deploy a new release. This approach is tedious and it leaves teams prone to delayed releases and application rollbacks.

    But with feature flags, developers can continuously add new features to an application without worrying that bugs within those features will lead to a rollback or a delayed release. If one feature has a problem, developers can simply turn it off using a feature flag, without having to do a new release.

    How SREs Can Benefit

    It’s not just developers whose lives are made easier—SREs, too, can leverage feature flags as a way of supporting the core SRE goal: Maximizing reliability.

    That’s because feature flags minimize the risk associated with fast-moving software development pipelines. They allow teams to experiment with new application functionality—which is important if they want to keep pushing out application enhancements to delight users—while at the same time providing a safeguard against buggy feature implementations.

    Imagine, for example, that you’re an SRE and you discover that the latest release of an application you support has a severe latency problem due to an issue with a new networking feature in the app. If that feature is controlled via a feature flag, you can remediate the issue very quickly by simply switching it off. You don’t have to wait for developers to fix the underlying code issue before you can solve the problem for your users.

    Limitations

    This doesn’t mean, of course, that feature flags are a cure-all against reliability problems.

    They won’t help you if the root cause of your reliability issue lies within your infrastructure rather than your software. If an app crashes because you misconfigured your Kubernetes clusters or you ran out of server capacity, for instance, you can’t simply turn the problem off with a feature flag.

    Note, too, that feature flags can only be applied to specific features. In other words, they can only control application functionality if developers deliberately created one to control that functionality before releasing the app. If you discover a bug that’s not managed by a feature flag, the only way to remediate the issue is to redeploy the application (or microservice, if the problem is limited to just one microservice).

    How SREs Can Encourage Feature Flag Use

    As an SRE, you most likely don’t have direct control over the software development process. You’ll therefore need to collaborate with developers to ensure that they build feature flags into their applications.

    Doing this is simple enough. Start by educating developers and making sure they understand the benefits. Be sure, too, to identify which features are most likely to create reliability issues, so that developers can prioritize creating feature flags for them. And consider factoring them into your incident response playbooks, which will ensure that incident response teams remember to take advantage of them when possible to remediate issues quickly.

  • Top Nine Skills for SREs to Master

    Top Nine Skills for SREs to Master

    It’s easy to talk at a high level about what site reliability engineers do: They ensure that IT systems achieve availability and performance requirements.

    But which skills, exactly, do SREs need to do to perform their jobs? That’s a more complicated question.

    To answer, let’s look at the top nine site reliability engineer skills that modern SREs (or aspiring SREs) should master. Although SRE skills may vary from one team to the next depending on the types of systems managed and the main types of reliability challenges faced, virtually all SREs need a core set of standard skills that allow them to understand and manage the type of complex, distributed systems they will have to support at the typical organization today.

    Here’s a breakdown of top SRE skills.

    Networking Expertise for SREs

    The network plays a pivotal role in connecting modern, distributed environments. As such, it’s often the culprit when something goes wrong—a lesson that Facebook, for example, learned when a networking problem brought down its entire global infrastructure.

    Situations like this are why SREs should master networking concepts. Even if their organization also employs networking engineers, site reliability engineers need a deep understanding of networking themselves to know when the network is the root cause of an incident and how to resolve network-caused issues effectively.

    Linux and Unix

    If you come from a Windows background but you want to be an SRE, there’s no getting around it: You’ll need to learn how to work with Linux and other Unix-like systems in addition to Windows.

    That’s because, even at organizations that don’t rely heavily on Linux servers, you’re likely to find that Linux and Unix concepts are deeply embedded within other systems that you have to work with. Most public cloud management tools follow the conventions of Linux CLI tools, for example. So do systems like Docker and Kubernetes, even if you run them in a Windows environment.

    Cloud Computing

    Like Linux and networking, cloud computing is another category of skill that modern SREs can’t live without.

    The reason why is almost self-explanatory: Around 90% of businesses use the cloud, and you can’t manage reliability for cloud environments very well if you don’t understand cloud architectures, cloud networking, cloud data storage, cloud observability and so on.

    CI/CD Pipelines

    SREs don’t typically help to develop software, but they nonetheless need a deep understanding of how software is written and deployed—which, at most organizations, today, is a process that happens via a CI/CD pipeline.

    It’s hard to engineer reliability if you don’t know how to address reliability problems that emerge from application source code or deployment processes. Understanding how CI/CD processes work and which tools drive them is key for virtually every SRE today.

    Quality Assurance and Software Testing Automation

    SREs also don’t usually help to test software pre-deployment. That task falls to developers or quality assurance engineers.

    Nonetheless, understanding how software is tested—and how to use test automation to speed tests and expand test coverage—is a vital SRE skill. After all, the more thoroughly and efficiently your team can test software, the greater your chances of catching reliability problems pre-deployment, when they are easier to fix and pose a much lower risk to the business.

    Security Engineering and Response

    Securing is another domain that SREs don’t “own,” but where they nonetheless require significant skills.

    Indeed, good reliability engineering makes security a priority, and vice versa. SREs who don’t understand security fundamentals are at risk of implementing reliability solutions that are effective from a reliability standpoint, but not necessarily secure.

    DevOps

    Although SREs are not DevOps engineers, SRE and DevOps are closely related domains. SREs at most organizations today will be expected to understand DevOps concepts and, in many cases, work alongside DevOps teams.

    So, plan to master DevOps skills as part of your site reliability engineer skills acquisition strategy.

    Incident Management

    Perhaps the single most important type of skill for SREs to learn is incident management. Although many roles may participate in incident response, SREs usually take the lead in organizing the incident response team, communicating with stakeholders and devising the best strategy for resolving each incident as quickly as possible.

    This means SREs should know how incident response roles are structured and understand incident response concepts. They should also be familiar with incident response platforms that automate the complex processes required to ensure rapid, effective incident resolution.

    Managing Postmortems

    In addition to overseeing incident response, SREs may be tasked with managing post mortems. Knowing how to run a postmortem—as well as when a postmortem is necessary, and when it makes sense to use a “blameless” postmortem approach—is an essential SRE skill.

    The list of SRE skills could certainly go on. Above are only the most fundamental types of skills SREs will need for most modern environments. But if you’re just starting out on your journey to becoming an SRE, the nine skill domains described above are a good place to begin acquiring the knowledge you’ll need to excel in an SRE career.

  • Service Level Objectives – Techstrong TV

    Service Level Objectives – Techstrong TV

    Service Level Objectives (SLOs) are being more widely discussed as they provide a clear path to delivering exceptional results by defining clear reliability goals. Kit provides updates on the community, use cases and best practices. The video and a transcript of the conversation are below.

    Recording:              This is Digital Anarchist.

     

    Alan Shimel:            Hey everyone. Welcome to another Techstrong TV segment. I’m really happy to have my friend Kit Merker all done up in his Seattle Kraken hat and his winter beard for the pacific northwest up there joining us today. Ken of course is the COO of Nobl9, a recent winner of the devops dozen awards as best new company on the devops space. Hey Kit. Welcome to Techstrong TV.

     

    Kit Merker:             Hey Alan. Great to see you.

     

    Shimel:            It’s good to see you again my friend. I’m thinking Kit we may actually see each other in person soon. I’m hoping.

     

    Merker:             Wouldn’t that be great? Hopefully somewhere sunny and I can get rid of the beard.

     

    Shimel:            Well I’ll tell you. I was just looking at airfare for Valencia, Spain for Cube Con in May. I don’t know if you guys are planning on being there for that one, if that’s like a CNCF thing you go to or what or – actually we have a road show coming up in Seattle at some point. I’m not sure when. We’ll talk about that.

     

    Merker:             Well in May I’m not sure about Cube Con but we will be having SLO Con and we’re doing it online again this year in May so that’s what I’m looking forward to.

     

    Shimel:            Yes. I think I saw that. What are the dates?

     

    Merker:             It’s May 9th through 12th. And that will be completely online and asynchronous and attend while you work just like last year. So we’ll have tons of speakers and all the fun from last year that we had and hopefully even bigger. The CFP is open right now so if you go to slowconf.com you can submit a talk. And it’s going to be exciting. And you guys covered the, were a media sponsor last year so hopefully we’ll have you back.

     

    Shimel:            Yes, we were.

     

    Merker:             Not to put you on the spot.

     

    Shimel:            I remember. We’ll talk about it.

     

    Merker:             All right.

     

    Shimel:            So that works out well because I think Cube Con if I’m not mistaken is like May 16th.

     

    Merker:             Yeah. We intentionally didn’t put it the same dates so.

     

    Shimel:            Yeah. So that will work. Cool man. Let’s talk a little Nobl9 though.  I guess we really shouldn’t. I’m assuming everyone knows who Nobl9 is. But I forget you’ve still only been around two, what, two, three years and at most. And not everyone may be familiar. Why don’t we back up and start there?

     

    Merker:             Sure.

     

    Shimel:            Tell people about Nobl9.

     

    Merker:             Of course. Yeah. So Nobl9 we make a service level objective platform, SLOs. And the whole idea of SLOs is about measuring your observability system and bringing new context, shared context so that you can make better decisions and better automation within your organization. So if you think about how you’re pulled in all these different directions, right? Some people on your team say we’ve got to ship features. Some people say if we don’t make it reliable and perfect then we’re going to hurt our reputation. If we don’t fix our tech deck we’re going to burn out all of our employees and we’ll have the great resignation to deal with. They’re all getting pulled in all these different directions.

     

    And what’s interesting about this is that we have actually more data about our software than we’ve ever had before. We have source control. We’ve got our AWS Cloud. We’ve got our Kubernetes cluster. We’ve got all these observability tools. We have so much data and yet we come to these different conclusions about what we should work on. And this is a big problem for so many organizations. So what Nobl9 is doing is we’re bringing some additional context. This is what’s really missing is people need to understand in an organization what are the risks to the system. What are the expectations that customers have and what are the tradeoffs we have to make in order to deliver the right kind of service to our customers?

     

    So what we’re doing with service level objectives in Nobl9 d is we collect all this observability data, we add this context and we drive automation and decisions so that whether you’re on call, you’re planning your future roadmap or trying to justify your investments you can see exactly what the impact is on the business and everybody can start pulling in the same direction. And that’s really what we’re doing. And like you said we’ve been around for a couple years. We now have a product in market. We both have kind of an entry level thing called Hydrogen, an enterprise class product that’s being used by some of the world’s largest enterprises and it’s very exciting and we’re having a great time.

     

    Shimel:            Very cool. Kit when we look over the evolution of the last couple years, right, ‘cause I remember with Nobl9, right, we first thought of it around well this is an SRE kind of thing. Right?

     

    Merker:             Yeah.

     

    Shimel:            And SREs early on were inherited a world of SLAs. Right? Where it was perform or a quick bullet to the back of the head will work.

     

    Merker:             Metaphorically speaking.

     

    Shimel:            But that wasn’t a very forgiving model or a model that would really kind of foster cooperation and team building and a lot of the principles behind devops, right, about how we get stuff done. It’s very hard to have a blameless postmortem when if you blew an SLA someone’s job is on the line. Right?

     

    Merker:             Well it’s really interesting because I think definitely in the sort of earlier I would say IT service model, right, where you have SLAs, especially if you have maybe outsource contracts. Right? SLAs became really a form of punishment. Right? One of the interesting things that I learned from sort of the Google SLA model is that there’s an assumption of reliability at Google, an assumption of infrastructure investment that’s not actually there in most enterprises. The SLOs that are prescribed by the SREs in the Google model are really trying to hold back over investment in infrastructure and this sort of gold plate infrastructure problem. Well most companies in the real world.

     

    Shimel:            It’s the other way.

     

    Merker:             It’s the other way around. Their issue is they can’t keep up with the reliability needs and the feature pressure is really the issue that they have. And so with this SLO model and being able to define them in fine grained ways across multiple data sources, being able to use it with automation. What changes is that you can start to put guardrails around these different conditions that people care about. You start to model the business. And this is really what we’ve got to get to. It’s about building the technology for digital business because all the businesses are digital now. Right?

     

    Shimel:            Right.

     

    Merker:             So you’re point about kind of like how you adopted sort of like devops method and bring this to the table, I mean really devops is trying to get better at building software, become more kind of diligent and efficient and make the right tradeoffs. It’s about ownership, right, a sense of responsibility but not punishment. And I think once people realize – actually I just had this customer from Flexera. We just did this testimonial video together. And he said it’s like once people realize that they weren’t building SLOs to point blame at them or punish them, that we were trying to do it to help them so that the people in the organization could see when they had problems, right, when they needed help.

     

    Then people really got open to this idea. This to me this is the game changer. If we can help organizations become more transparent, to be able to do more through APIs instead of meetings if you know what I mean. Right? Like using the data and the systems to help make these better decisions. This is what changes culture. A lot of people want to talk about reliability culture and devops culture and agile culture but that’s such a scary idea. The way I like to think about this and what we’ve talked about with our customers and community members is it’s about how you make decisions together. Right?

     

    And if you can make lots of decisions in a way that is supportive of the strategy, supportive of the business and also the realities of the pressures that we’re on as humans. And let’s not forget that everybody is going through a hard time and maybe not totally happy being on call and all these other things. Taking all that into account and making a decision together this is what becomes culture, not the other way around. Right? You don’t change culture first to get the decisions. You start making good decisions and then the culture emerges from that. And that’s I think a really important idea.

     

    Shimel:            I agree with you 100 percent. No doubt about it. And it is. But when we focus on devops being about delivering software and clearly that’s the goal. Right? Delivering software better, faster, higher quality. In order to do that you need just that whole culture thing. And some people say oh culture is BS. It’s not. Having a team that works together, having goals that are clearly enunciated, right, we’re all trying to do – you understand what the guy or gal next to you is doing. Right? And why they’re doing it and how you fit into it. That’s all part of that. All of those things help you get software better, faster, higher quality. And that’s where it is.

     

     What’s interesting about Nobl9 Kit from where I sit, right, because I sit – I mean beyond being on different sides of the continent I sit in a different chair than you. I get to look at the whole market without a horse in the race so to speak. You guys have done a really great job of defining and owning SLOs. Right? I think for a lot of people in the marketplace, right, SLOs aren’t necessarily synonymous with Nobl9. But you’ve probably done more to promote SLOs and their use and why they’re beneficial than anybody else in market.

     

    Merker:             Well first of all I appreciate that a lot. Personally I think it’s a reflection of the community. I think it’s a reflection of a nerve that we hit at some level. Because to your point, right, goals that are well enunciated and it’s funny to me because I think oftentimes we think of goals as things that are kind of squishy. Right? We set stretch goals – oh sorry. I thought I muted everything.

     

    Shimel:            It’s never all muted.

     

    Merker:             We’re all muted. We want to set goals. Sometimes they’re a little squishy is what I was saying. So it’s like maybe you have OKRs. Maybe you have stretch goals. Maybe you have some different perception of what the goal is for your service. And what SLOs are doing is turning it into pretty much the most precise type of goal you could imagine which is code, right? Like actually describing it in code and something that’s checked in where it’s the truth. There’s no arguing about what it is. Now we can change it, right? We can always change our minds about what the goal is. But at any given point in time there is a goal that’s described and there’s a set of actions associated with that goal. And to me this is exactly what is so attractive to people in the market about SLOs because they’re no longer guessing.

     

    They’re no longer sending each other Slacks and emails and writing documents and putting it in dashboards. They have a very precise way of describing what they want and expect from their service that takes in account the risks and tradeoffs and customer expectations that they have for the service and then allowing the team to engineer to that goal, allowing the business to understand the tradeoffs. Right? Hey. If you want the service to be faster we’re going to have to spend more to buy more computing power. Right? Hey. If you want these features we’re going to have to put these other reliability concerns on hold and vice versa. This is something that brings visibility to all the little engineering decisions that are made on a daily basis and then aggregates them up to the size of an entire enterprise ‘cause it’s birds eye view of how the enterprise functions and doesn’t limit them to a prescriptive way. ‘Cause I think this is one of the other big challenges.

     

    When I look at the observability space or anybody’s infrastructure space is trying to bring a prescriptive guidance. I think today’s world is about the right tool for the job. Right? You’ve got to fit inside of an ecosystem whether that’s what language you use, what CICD system you use, what package manager you use, what source control you use. All these things are choices that engineering teams have to make for themselves. And you can’t necessarily dictate that Team A and Team B are going to agree on that stack if you will. Right? You give them some flexibility. But what you can get them to agree on is the concept of the service level objective. Like what should my service provide to customers in terms of availability, latency, performance, throughput, whatever the characteristics are that deliver that end to end service.

     

    This is what becomes almost the contract because if I know that I’m using your service and you’re going to give me x number of latency over y number of nines then I can build my system to account for it. I can decide what my retry strategy is, what my caching strategy is. Right? I can design and engineer a system that takes into account what your service says it’s going to do. Well that level of clarity allows engineering teams to operate in a completely fundamentally different way because now they aren’t guessing and they’re not hoping and they’re not wishing. Right? They’re working toward a clearly articulated goal. And I think this is what’s so important. So now the whole idea of us promoting this concept. I mean we’re just like we’re in love with SLOs. I mean I don’t know how else to explain that. Like we’re just like losing our mind about how amazing it is.

     

    But it is really about the community. We now have literally thousands of people that are involved in the SLO, the SLO Conf community. We have the monthly meetups. The conference as I mentioned will be in May. And we’re expecting hopefully about 5,000 people to show up this year. Last year we had 2,200 registrations. So we’re pushing hard for this but based on what we’ve seen in the market and the companies that are literally deciding if they’re going to build or buy an SLO platform right now.

     

    There is a huge groundswell of interest in this concept to take their observability systems, their operation systems, reliability systems to the next level. And all the practices and tools and techniques and tricks everybody wants to share it and learn together which is I think the most exciting thing. And it fits in with so many other areas, incident response and chaos engineering and how we do monitoring and logs. All this stuff kind of fits into this whole story about SLOs. Even finops, right, even like cost management, all this stuff fits into this same story. So yeah. It’s kind of overwhelming to be honest how much the market seems to have appeared and materialized over the last couple of years with SLOs.

     

    Shimel:            Cool man. Hey Kit, we’re up on the clock here. So you mentioned SLO Con and SLOconf with an F dot com. Right?

     

    Merker:             That’s right.

     

    Shimel:            Right.

     

    Merker:             Yep.

     

    Shimel:            For people who want to maybe submit a talk or just find out more about attending this year’s virtual conference in early May. And then for people wanting to get more information on Nobl9 is it Nobl9 dot com or IO? I forget.

     

    Merker:             Nobl9.com. Yeah. N-O-B-L number 9, dot com.

     

    Shimel:            That’s it. Hey Kit, man. It’s great seeing you. I hope to see you in person sooner than later here. But keep doing what you’re doing man and congratulations on the devops dozen award as well.

     

    Merker:             Thank you so much Alan and hopefully we’ll catch a hockey game together.

     

    Shimel:            Come on down. Actually I don’t have Seattle Sounder tickets when they’re playing the Panthers but we’ll look into it. I’ll talk to you later man.

     

    Merker:             All right..

     

    Shimel:            Great seeing you.

     

    Merker:             All right. Take care Alan.

     

    Shimel:            All right. Bye bye.

     

    Merker:             Bye.

     

    [End of Audio]

  • Reflections: A DevOps Christmas Carol

    Reflections: A DevOps Christmas Carol

    “Marley was dead, to begin with.” So begins my DevOps Christmas Carol. In this reflection, I am visited by the ghosts of DevOps past, present and future. I hope that, after hearing my story, you will have a better understanding of where DevOps is going based upon where it came from and where it is now.

    The Ghost of DevOps Past

    The ghost of DevOps past took me back to Las Vegas, 2012. I shared a bottle (or two) of wine with Gene Kim, Gene was talking about a manuscript for a book he was working on based on The Goal by Eliyahu Goldratt. It was a digital story of resource constraints and the way forward based on something called DevOps. I was fascinated by this new way of thinking about development and operations while musing about what a great thing that could be for security, which was my thing then. A whole new world opened up to me as I met and learned about the work of people like Patrick Dubois, John Willis, Jez Humble, Damon Edwards and more.

    I was taken to Boulder, Colorado where Foundry Group and Techstars were putting on a symposium to make sure their portfolio companies were smart about DevOps. The CTO of Rally software did a presentation on Agile and DevOps and, again, it was like looking at the IT world from a whole new vantage point. It was after speaking to some of my friends, Rajat Bhargava and Brad Feld, that the idea for what became devopsy.kinsta.cloud was born.

    What a golden time that was. DevOps was an elusive thing. No manifesto, no official definition. Some people said it was mythical and not real. Others said it was just for startups born in the cloud. Still others said it was just for large enterprises. DevOps, it seemed, could be everywhere and nowhere, all at once.

    Looking back, it was an exhilarating time, with philosophical arguments vigorously debated. And over the weeks, months and years that followed, the idea of DevOps became a real thing, the benefits it provided revealed themselves and a set of best practices emerged. My partners at the DevOps Institute and I tried to capture these emerging best practices and build courseware and certifications around them. Oh, my—what a sh*tstorm that created. The idea of being certified in DevOps was scoffed at, derided, spit on and ridiculed. But that, too, eventually passed. Looking back at this part of my trip, I realized that while it all seemed idealistic and exciting and despite the ups and downs and bumpy road, we were always moving forward.

    The Ghost of DevOps Present

    Next, the ghost of DevOps present visited me. In this dream, DevOps had begun to blend into the woodwork. For the most part, it was an accepted way of developing and operating software. Many principles such as CI/CD, continuous testing, SRE and DevSecOps were mainstream. While not 100% fully adopted, the majority of organizations used some DevOps patterns in their business.

    It had become so mundane, in fact, that people began to ask, “Has DevOps hit the wall?” “Has DevOps passed its peak?” and “What’s next for DevOps?” While this may sound ominous, it is actually par for the course in the adoption cycles of new technologies. As certain as the Gartner hype cycle, technology that has settled into the stack seems to be the way of things. So as much as the ghost of DevOps present tried to frighten me, I was unafraid. Adoption was proceeding just as expected. It was firmly entrenched as part of digital transformation initiatives. Cloud-native and DevSecOps also were becoming more and more accepted.

    The Ghost of DevOps Future

    Next, I was visited by the ghost of DevOps future. She showed me a glimpse of what the future held. In this future version, SREs worked side-by-side with DevOps teams. There was a smooth handoff at the time of deployment ensuring that apps behaved the way they were intended. Automated feedback loops allowed developers to incorporate lessons learned into future versions. 

    Low-code/no-code solutions designed for developers allowed for even faster, automated deployments. AI-enabled applications let testing keep pace with development, more automation enabled even faster deployments. 

    AI also made observability more than just a spectator sport. Actionable intelligence that was wisely acted upon was the norm in this future. Of course, security was still an issue. There were even more regulations and compliance requirements. The bad guys were as notorious as ever. It seems even a DevOps future where security was everyone’s responsibility could not fully solve our security issues. But, overall, progress had been made.

    DevOps teams were deploying directly from Git onto cloud-native stacks that started with Kubernetes but didn’t end there. There were service meshes and APIs connecting everything. Serverless platforms were offered by most vendors. Overall, no one wanted to be locked into one cloud provider and picked the right cloud for each project. Indeed, DevOps in the future seemed on the verge of recognizing and achieving many of its aspirations.

    I awoke and felt good about everything that was happening with DevOps. In the words of Tiny Tim from that other Christmas carol, “God bless us, every one!”

  • Kentik Open Sources Five Projects to Advance Network Observability

    Kentik Open Sources Five Projects to Advance Network Observability

    Kentik today unfurled a developer hub, dubbed Kentik Labs, through which it is making five open source projects available as part of an effort to advance observability across network operations (NetOps).

    The five projects are:

    kTranslate: A system for pulling and pushing network data that is the basis of the Kentik Firehose integration the company created for the New Relic observability platform

    NetDiag: Scalable, asynchronous implementations of low-level network diagnostics

    Convis: Example code showing how to attribute process and container information to network traffic via the Linux kernel using eBPF

    kProbe: A high-performance host and sensor network probe (PCAP)

    Grafana App: A real-time tool for ingesting and querying network data

    Nick Stinemates, vice president of business development at Kentik and co-chair of Kentik Labs, said the goal is to enable DevOps teams to incorporate NetOps within the workflows already being employed to programmatically manage the rest of IT.

    Stinemates noted that, as IT environments become more distributed, managing NetOps has become more challenging. If there is a failure, it can be difficult for DevOps teams to determine if the issue is due to application code, the underlying infrastructure or some other root cause. Kentik Labs is committed to providing tools that will help DevOps teams reduce their mean time to recovery (MTTR), he added.

    Most NetOps teams increasingly rely on software to manage distributed networks either in the form of graphical tools provided by router and switch vendors or by using the Python programming language and the open source Ansible automation framework that enables IT teams to declaratively manage IT infrastructure.

    As the teams that manage network operations gain more proficiency with those tools, it becomes simpler to incorporate them within the construct of a larger DevOps team. While DevOps teams have been programmatically managing IT infrastructure-as-code (IaC) for years, network management has largely remained in a separate silo. However, as more applications are deployed in the cloud, it’s clear NetOps needs to become more integrated with the rest of the workflows employed to deploy and manage a highly distributed IT environment, said Stinemates.

    Less clear is to what degree the networks that make up a hybrid cloud computing environment will continue to be managed by network administrators versus site reliability engineers (SREs) that are starting to assume responsibility for managing compute and storage.

    Regardless of who manages the network, the next generation of microservices-based applications that are dependent on application programming interfaces (APIs) to call various services are much more latency-sensitive than previous generations of applications. Many of those applications will regularly experience performance issues due to networking issues that are often not readily apparent to DevOps teams. The goal should be to, at the very least, arm those DevOps teams with the tools required to confirm the source of those issues before a “war room” is convened to determine how to address the issue.

  • How to Build an Options-Based Observability Strategy

    How to Build an Options-Based Observability Strategy

    The global pandemic forced companies to accelerate their digital transformations, pushing them toward new technologies and architectures as they struggled to adapt to a changing world. Companies wanted fast development and deployment and were happy to trade off efficiency, reliability and sometimes, security, to pursue their goals. Now that the pandemic is receding in many parts of the world, the priorities are rebalancing as companies grapple with choices they made under fire.

    Observability has emerged as a way for enterprises to better understand sprawling infrastructure and application environments. According to Gartner, “Observability enables organizations to reduce the time it takes to identify the root cause of performance-impacting problems. In particular, and in contrast to traditional monitoring, operators can freely interrogate data post-hoc without the need to pre-program dashboards.”

    This sounds promising and the market responded quickly, but predictably. Large and monolithic monitoring platforms embraced the term and added new features, making them even larger and more monolithic. These monolithic platforms represent a failed choice-based strategy. They assume that all observability data is collected, centralized, integrated and only analyzed by a single tool. Enterprises following a choice-based observability strategy find themselves locked-in to multi-year investments without the flexibility to react to evolving technologies.

    While this has been good for vendors in the incumbent monitoring space, end users haven’t benefited from these enormous platform deployments because the choice-based strategy is built on the faulty assumption that a single pane of glass serves all of your monitoring and observability audiences, from SREs to DevOps and SecOps.

    A choice-based strategy also forces customers into limited options in the future as technology continues to evolve and requirements change. Defaulting to choice-based strategies also leaves little leverage for contract negotiations, data integrations and data access.

    Infrastructure and operations teams need to take a different approach to their observability strategy. Instead of focusing on buying products and making choices, teams need to shift their thinking to creating options. The choices you make today limit your options tomorrow. Tomorrow is unpredictable and uncertain. Taking an options-based approach allows you to be flexible and adaptable today and into the future.

    An options-based observability strategy is built on three pillars:

    • Establishing your observability strategy principles
    • Building composable environments
    • Staging your commitments

    Let’s look at each of these.

    Establish the Principles of Your Observability Strategy

    When asked about their observability strategy, many often leap into detailed plans covering every imaginable use case. They talk about the current state of their infrastructure and how things will be at the end of three or five years, including the skills and functions they’ll need in that time. If you’ve read the above, you’ll recognize this as a choice-based strategy: All the choices they need to make over the next several years are already laid out. This leaves little room for adaptation and options as the world evolves around you.

    Instead, think of your observability strategy as a single principle, or a collection of principles, guiding your observability strategy. Let’s say your core principle is that observability is a key enabler of operational efficiency across SREs and DevOps. We can build on that operational efficiency principle to define what key elements of our strategy has to include:

    • Data sources have utility-like accessibility; always on and available across tools.
    • Metrics are primarily driven by total cost of ownership and optimization.
    • Governance is centralized to the team distributing data to downstream systems.

    These elements are different if your core principle is something like customer intimacy or risk management.

    With principles and strategy in hand, we can focus on the next phase: Composing the observability environment.

    Compose your observability environment

    In an options-based strategy, the key success factor is avoiding large ecosystems that limit flexibility. Many vendors take a “give us everything” approach to observability. These approaches appear to start and end at the destination platform, but they begin even earlier, at data origination. It’s common to see agents only sending data to one destination, limiting the usefulness of the data by locking it into a silo.

    Staying modular means selecting tooling that abstracts the data sources from destinations and puts the power back into your hands. By using an abstraction like an observability pipeline, you can send data from one source to multiple destinations, while managing that data in any way you see fit. For example, this is where your centralized governance can occur.

    When new tools are introduced, upgraded or deprecated, an observability pipeline insulates the rest of your environment and users from those changes. A modular approach also gives you more power in contract negotiations with your other vendors because you’re less reliant on one vendor for all of your observability needs.

    Stage Your Commitments

    A choice-based observability strategy is a large commitment, often taking years. Once you start, stopping becomes impossible. That’s simply unacceptable in today’s environment of uncertainty. Choice-based strategies limit your ability to test, experiment, and learn. At the same time, our infrastructure is critical and it must be reliable.

    An options-preserving approach may involve experimenting with freemium products; only paying once the product’s value has been proven. Cloud-based options are another consideration since they are commonly subscription-based and offer easy try-before-you-buy purchasing styles. Even better, cloud-based options don’t require using your own infrastructure to test new options.

    Every IT team feels the need to react and respond to demands from the business. That urgency to make choices, to solve the immediate problem, can have long-term implications for future flexibility and agility. By structuring your observability strategy around a long-term vision based on preserving options, your team can remain adaptable as technologies and business requirements evolve.

  • Fixing Risk Sharing With Observability

    Fixing Risk Sharing With Observability

    Incentives are mismatched among SREs, SecOps, and application developers. These mismatches create challenges around how and what information is shared across siloed teams. This asymmetrical information creates a moral hazard where one team can shift deployment risk to another team, with no accountability back to the originating team. Risk shifting results in unstable applications, inefficient infrastructure, security issues and poor customer experience. All of that impacts your company’s bottom line.

    Closing the Information Gap

    Observability is positioned as a way to fix the information gap between these disparate groups. Observable systems allow users to ask questions about their behavior in an open-ended way, unlike more rigid monitoring systems. SREs and SecOps teams are already reaping benefits from observability, but mismatched incentives limit its effectiveness. Meeting the observability expectations of IT leaders requires pervasive instrumentation across applications, infrastructure and third-party software. Delivering that level of instrumentation has remained out of reach due to incentive mismatches, as well as human and infrastructure costs.

    The main incentive for developers is shipping code quickly at a reasonable level of quality, meaning a low bug count. This contrasts with SREs, who are incentivized around uptime, performance and efficiency, while the incentives for SecOps teams are around risk reduction and breach mitigation.

    risk sharing observability

    Since SREs and SecOps teams don’t have insight into what developers have changed, it’s easy for performance or security problems to get thrown over the fence to operations. A fresh code push might only include minor changes and be harmless to existing operations. Or, it could replace large chunks of logic across the entire codebase, including adding calls to external and third-party applications. To the intrepid SRE and SecOps teams, one change looks like any other. They don’t get a peek behind the curtain because of the mismatched incentives. DevOps wants to deploy quickly. Waiting on approval from other teams slows down deployments, so comprehensive reviews don’t happen. This doesn’t mean DevOps teams are intentionally trying to sabotage partner teams. They’re simply acting in their own interests based on their incentives.

    The challenge is that one party, the developers, has more information than other parties. That information asymmetry is what creates unbalanced risk sharing. Coping with information asymmetry has led to all kinds of new collaborative models, starting with DevOps and evolving into DevSecOps and other permutations like BizDevSecOps.

    True collaboration has been hard to come by. Early DevOps efforts are often successful, but scaling beyond five to seven teams is difficult because teams lack the breadth of experience in IT operations or the SRE capacity to staff multiple product teams. The change velocity DevOps teams can achieve is often far greater than SREs and SecOps can absorb, making information asymmetry worse.

    If teams can’t maintain high levels of collaboration and communication, another option must be developed. Observability practices, like collecting all events, metrics, traces and logs, allow SREs and SecOps teams to interrogate applications about their behavior without knowing which questions they want to ask ahead of time. However, observability only works if applications, and the infrastructure they rely on, are instrumented. This creates another problem: who does the instrumentation?

    The expectation is DevOps teams embed instrumentation into their code as part of the development process. While that’s a nice idea, there are four reasons this falls short. First, the quality of instrumentation varies. Many log statements are terse and only understandable by the developer that wrote them. The message “In function xyx123!” isn’t helpful to an SRE digging into a performance problem that cropped up in the latest release.

    Second, instrumentation libraries vary by implementation, giving inconsistent results across language bindings. OpenTelemetry is trying to improve this, but its progress is slow and still requires developers to do more work that, if we’re honest, doesn’t benefit them; it benefits SREs and SecOps. Now we’re back to those pesky mismatched incentives and information asymmetry problems again.

    The third problem with instrumentation is the volume of data. Each instrumented application can produce terabytes of data each day. When you have robust instrumentation, the amount of data can be overwhelming, and extremely costly to analyze and store.

    Lastly, instrumentation is isolated to the code your team wrote. That represents a fraction of the code you rely on. Vendor-provided services and APIs remain a black box, limiting your observability into those components.

    Resolving information asymmetry across teams requires two things:

    1. Pervasive, pluggable instrumentation capabilities for all code that doesn’t require developer involvement, and
    2. An observability pipeline to filter, redact and enrich data, then route it to your analytics platform of choice.

    Pervasive Instrumentation

    Operations teams need instrumentation without having to go back to developers and beg them to add it to existing code. They need to turn it on and off as needed, and they need readily consumable data. They also need every piece of data they can get, including packet payloads, insight into encrypted data and so on. This goes well beyond what’s possible with today’s instrumentation options.

    AppScope, a newly released open source project, is a new take on instrumentation. AppScope interposes itself between application threads and system libraries, tracking things like file system access, network and HTTP activity, as well as CPU and process activity. It also provides payload data, and because it sits between the application and encryption libraries, it also gives access to users’ cleartext data. Because it works with any Linux binary, SREs and operations teams can instrument anything, even code they didn’t write.

    Observability Pipeline

    The challenge with pervasive instrumentation is dealing with all the data generated by applications. Network, file system, and other system data can easily swamp destination APM and log analytics platforms, driving up licensing and infrastructure costs. While instrumentation data is vital to rebalancing risk sharing in organizations, you need a way to manage that data intelligently to get value out of it. This is where the observability pipeline comes in.

    An observability pipeline sits between the sources of observability data and its target destinations. By acting as a strategic control point, the pipeline gives users control over how data is formatted, filtered, enriched, and redacted before it is routed to one or more destinations.

    Observability pipelines help SREs and operations teams deal with the flood of instrumentation data by routing low value data to low-cost storage, like S3, while higher value information lands in APM and log analytics tools. Later, if you decide you need the data stored in S3 to add more context to your analysis, you can replay it back through the pipeline and enrich your data set.

    Shifting business priorities are accelerating the pace of technology deployment across organizations, and teams are pressured to deliver faster. This faster pace of delivery pushes deployment risk to operations and security teams that lack visibility into changes developers have made across complex distributed systems. This information asymmetry also increases risk to the business as applications become less predictable and reliable over time.

    Traditional methods of resolving the information mismatch haven’t worked because incentives across teams aren’t aligned. Adopting pervasive instrumentation and observability practices will give SREs and operations teams critical visibility into rapidly changing application and infrastructure environments, without disrupting the developer experience and process.

  • Incident Resolution for Remote Teams

    Incident Resolution for Remote Teams

    People working in IT support and incident management right now are faced with unusual difficulties supporting large remote workforces and managing unpredictable workloads. On Reddit, system admins and other IT pros are bemoaning the hiccups and hassles of working in isolation while trying to resolve issues and maintain high SLAs. You can’t go grab your indispensable SME for troubleshooting, because that person is also home and inundated with messages and alerts from many different tools.

    Networks, VPNs and servers are being hammered extra hard right now from higher usage of SaaS apps and collaboration tools. User expectations are even more grandiose than normal: The home-based IT environment is a lifeline to whatever remains of human interaction. IT organizations are expected to make applications and services work flawlessly–overnight. Undoubtedly, IT pros are working with assiduity to keep the wheels of work moving for their employers during a time of crisis. Yet, working longer hours isn’t the answer. We need new ways of working—ones which may prove better than the current status quo for the long term.

    In many organizations, the process of incident management and resolution begins with people looking at screens full of metrics. You see a spike in something, such as network I/O, and immediately begin to investigate it, assuming the worst. A more adverse scenario is when users call out problems on Twitter, Reddit or other popular tech communities, creating a noxious PR situation which IT and the business must swiftly respond to and manage. This ad hoc, reactive fire drill takes time, and resolving that particular issue may not be the most important item of the day. In software development, it’s akin to the bad practice of designing the architecture of a system first and then the user experience.

    Given everything that’s at stake in the foreseeable future of stay-at-home mandates, SREs and IT support technicians need to flip the incident management process. Rather than beginning with the metrics and working backwards to uncover the problem, if there even is one, start by looking for the most important problems to solve. Then, discover the resources and components which support that end-user process and finally, analyze the metrics which are most relevant. For example:

    1. A B-to-C provider of a streaming video service is now seeing a 200% increase in daytime traffic. This is resulting in sluggish load times for users and glitches. 
    2. IT has tagged the components beneath that service, so IT operators know exactly which VMs, databases and load balancers come into play. 
    3. Next, the operations team can identify the most relevant metrics, for instance CPU idle time, to measure and take steps to correct. 

    The Problem-First Approach

    It sounds so simple: flip the process. Yet, this is not the case. IT operators and SREs are not accustomed to thinking in terms of which problems to solve or prevent. Being proactive or predictive hasn’t been the common workflow.

    Here is how IT operations and DevOps teams can take a proactive approach to managing the availability and health of enterprise services.

    Coordinate Across Departments

    Identifying the most critical problems to solve right now is the hardest step and the one which will take the longest. Common problems these days are related to cost and scale: How do I right-size my environment to reduce costs or handle more workloads? Broaden your field of inputs to narrow down to the specific issue, such as slow page loads on the financial reporting website. Product managers, account managers and business unit leads who are the closest to the customer experience can deliver feedback on the top issues affecting customer/user satisfaction. Teams should also review recent support tickets to identify common themes of pain.

    Fine-Tune Your Approach to Metrics

    The Utilization Saturation and Errors (USE) Method is one way to approach the problem-first incident management process. As detailed by Brendan Gregg, a senior performance architect at Netflix, this methodology begins by posing questions, seeking answers and working backwards to the metrics. For each resource that you want to measure, identify three metrics: one for utilization, one for saturation and one for errors.

    “The USE Method has made you aware of what you didn’t check: What were once unknown-unknowns are now known-unknowns,” Gregg explained.

    Standardize Workflows

    Create a common process for incident management across all your teams. Without the advantage of having almost everyone in the same room to huddle together in an ad hoc fashion when big issues crop up, it’s imperative to institute clear steps and roles. Doing so will prevent the frustrations, confusion and oversights that needlessly delay resolution. Since most incidents are composed of multiple contributing factors, teams need to adopt a small number of user-friendly tools to document and organize the information. At our company, we’re now using tools such as Miro, an online whiteboard application, to replace our physical whiteboarding sessions. Of course, there’s also Zoom, Slack, Jira and a host of other cloud-based tools already in place at many organizations. Mandate which tools everyone should use, with some guidelines on how to use them.

    Increase Automation

    In some organizations, scaling requirements in response to demand have increased tenfold. Automation is playing a critical role now; moving away from a web GUI, for example, is more scalable and aligns with modern tools such as Chef and Puppet. User tickets can be auto generated, for instance, from emails and linked to code management systems like GitHub. Modern development and operations teams are also expanding automation in unit testing and provisioning. 

    Watch for Burnout

    Whether there’s more work and/or a need to fill the hours during long quarantine days, many software engineers, testers and architects are working longer days right now. Yet, exhaustion and burnout can lead to errors and oversights along with low morale. It’s up to managers to make sure that employees are taking breaks, working reasonable days and having the time and energy to attend to personal needs.

  • xMatters Incorporates Stackdriver APM Metrics from Google

    xMatters Incorporates Stackdriver APM Metrics from Google

    As a set of application monitoring services on the Google Cloud Platform (GCP) for driving DevOps processes, Stackdriver has been steadily gaining adherents, especially since Google extended the reach of that service into the realm of on-premises systems. Today xMatters, a provider of incident management tools, announced it is integrating alerts and data generated by Stackdriver into its workflow processes.

    Company CTO Abbas Haider Ali said the xMatters incident management software will now automatically relay Stackdriver data to the correct people and systems to help coordinate and resolve incidents faster. Stackdriver insights will be embedded into notifications, collaboration invitations and service management tickets, said Haider Ali.

    The goal, he said, is to provide DevOps teams with more context concerning specific issues directly within the user interface employed to access the incident management platform.

    Google, via Stackdriver, not only provides access to a tool for application performance management (APM), it also effectively embeds best practices for site reliability engineers (SREs) inside the service. While the best practices defined by Google are highly opinionated, they are starting to gain traction as a specific methodology from implementing a DevOps process, said Haider Ali.

    Support for Stackdriver follows the recent release of an update to xMatters that added an Event Flood Control capability to suppress similar requests in close succession as part of an effort to reduce the level of noise generated by a single incident. The company also added an Unlimited Event Visibility feature allows organizations to unlock insights from past incidents to identify a potential root cause of a problem faster.

    Of course, while Google may have invented the SRE concept, it is clear other cloud service providers are coming up with their own flavors and many internal IT operations teams will likely create their own equivalents. But as the SRE concept continues to gain traction, the rate at which DevOps processes are adopted enterprisewide should increase. The challenge most enterprise IT organizations face today is their internal teams are largely made up of administrators rather than engineer that know how to programmatically manage infrastructure as code. Many of those organizations are either hiring former developers to become SREs, or investing in training to up level the skills of their existing IT administrators.

    But as more organizations embrace SRE concepts many of them will be looking to create closed-loop approaches to DevOps that enables them to significantly reduce the amount of time between when an issue or incident is discovered, and it gets assigned within a continuous integration/continuous deployment (CI/CD) environment to be programmatically addressed.

    In the meantime, any gaps that may have existed between DevOps processes and incident management systems are continuing to narrow. Not every organization necessarily has to hire SREs to achieve that goal. But those organizations that can afford to hire and retain SREs should expect they will insist on incorporating incident management with the context of a larger set of sophisticated DevOps processes.

    — Mike Vizard