devopsy.kinsta.cloud

Tag: troubleshooting

  • Don’t Hire for Product Expertise

    Don’t Hire for Product Expertise

    We all know the jokes about talent recruitment that “requires a century of experience in AI/ML.” With the number of layoffs, no doubt this type of approach will get worse. But that is not generally a tech problem; it is an HR problem–the problem of having people hire in tech that aren’t active in or knowledgeable about tech.

    The real concern that is very much a tech problem is hiring for product and/or technology. I will say outright: Stop doing that except in very special circumstances. Hire for the ability to find information. One of the other things we all joke about is increasingly convoluted interviews where the hiring company wants you to solve their problems in a fraction of the time it would actually take them to do it. Again, if you’re a hiring company doing this, knock it off.

    IT is an ever-evolving field, more so once Agile and DevOps are taken into account. Indeed, we would not be far off if we renamed the field Continuous Evolution, because tomorrow is guaranteed to hold different solutions, no matter what. The idea that requiring experience in Grafana or mastery of Python is an indicator of long-term success is downright farcical. There are positions where this is tolerable–if a position is for maintenance of a traditional app that is not going anywhere and is not getting massive updates, then a list of what tools are desired for the role is easy to build and will be lasting. But I would argue that hiring specifically for that application’s needs is not in the best interest of the organization as a whole, because that person is not portable and adaptable. For modern development, particularly of new products that are just now being architected, the needs are going to change in the process, so claiming you want X years of experience in Spring (when it may use a different framework by deployment time) is not productive.

    As we move into belt-tightening yet again, employees who can do the job and know how to look up new information will be worth their weight in gold. Having a person who can work on existing applications and drive innovation in future applications all in one body is important. Giving them a chance to exercise that drive to learn is also important, but is a separate issue. None of the areas I cover–applications, operations and security–are standing still or even coming close to standing still. The need for people who can go find out what is needed is huge.

    And while the internet is a horrendous mix of good and bad, the ability to find information—and even solutions to problems—is undeniably better than it ever has been. Having people skilled at using that information resource is the key to moving forward.

    So, find people that can solve problems given a modern context. Hiring for technology will solve today’s problems, but problems are not restricted to today, so solve ongoing problems. Gate for basic tech knowledge and skills, then evaluate for the ability to find answers. A far better use of everyone’s time during interviews would be—using email as an example—“Hosted Exchange just went down. Walk me through your troubleshooting steps. You have a couple minutes.” Or—using development as an example—“We just discovered that an OSS library we use has unacceptable vulnerabilities. Walk me through how you would find instances of it and what steps you would take to find replacements. Compliance is breathing down our neck, you have a couple of minutes to show us.” You get to see how they approach likely problems, they get to ask about the environment. Far better than “Show us all the steps to get BGP routing configured,” or “Develop an entire B-Tree while we watch,” questions. Certainly better than any question that ends with “And don’t use the internet.” Are you kidding? Can we talk about the field and what year it is?

    As I’ve said off and on over the years, you want the best coworkers you can get or the best employees you can get. Either way, hiring for expertise with a given tech product is not the best route to that goal. You rock—so get people with you who do, too. It’s not going to get less complex, so have peeps that can keep up and carry a load, no matter what tomorrow’s technology looks like.

  • A Cause for Complaint

    A Cause for Complaint

  • Learning Production Troubleshooting Techniques in Microarchitectures

    Learning Production Troubleshooting Techniques in Microarchitectures

    One of the great things about our industry is the ability for anyone to get started doing development with a minimal amount of tools. Anyone can learn to do development with just a computer of some sort, a text editor, a tutorial and their brain.

    In fact, the initial basics of how to think through a development problem don’t require a computer at all, reducing the cost of getting started even further. Having such an easy way to get started that doesn’t require a lot of tools lowers the bar for entry, enabling more people to join in on development teams and contribute new ideas and perspectives, which enriches the entire industry.

    Getting into containerization and building modern microarchitectures is a slightly higher bar, but only slightly higher. There are a number of wonderful tutorials out there for getting started with Kubernetes, Docker, OpenShift Origin and other free and open-source tools.

    The only differences in getting started between general software development and microarchitectures specifically are: needing a computer that has the processing power and memory to run containers, and enough of a grasp of a development language to have a basic application to play with. Even that last difference isn’t as crucial because there are so many sample applications people have built for the express purpose of teaching new folks to deploy microservices onto various systems.

    However, when it comes to learning from the DevOps, SRE or ops side of managing Kubernetes deployments or other modern architectures, there’s no solid free tutorial I could find that dumped a bunch of broken sandboxes in front of the student with the express purpose of building their troubleshooting knowledge.

    To gain insight into understanding how to bring microarchitectures back from failures or fall-overs requires access to larger applications and clusters and more senior, or more experienced, mentors–unless you are lucky enough to be able to build large interconnected applications and send traffic to them regularly enough to cause failures.

    This lack of strong tutorials and a requirement for more resources makes breaking into DevOps or SRE work on microarchitectures particularly difficult if no one is willing to take a chance on a less experienced candidate. I want to see that change. We are missing out on hearing from new voices, people who may look at a problem from a different perspective or who may have unique solutions to problems we all face.

    I decided to create a repository full of deliberately broken Kubernetes builds for new practitioners to build progressively more advanced troubleshooting skills. The basic idea is to build a pyramid of environments. The base level should be easy to understand and figure out. These environments will be bare-bones simple, with only one cause for failure.

    We need to build the baseline recognition of common causes in isolation. Many of us that have debugged production systems have an almost unconscious grasp of when a common cause has appeared simply because we recognize that faint signature in the pattern of the data we’re analyzing. If you’ve never seen that signature, though, and you don’t have a mentor with you on these debugging challenges, learning to find that signal in the haystack of your data is much harder. These initial environments should make it easier to learn what those signals look like.

    The next level of environments would have some additional factors unrelated to the actual problem causing additional logs, additional spikes in traffic or additional activity. In short, these environments would have noise. We can add another layer of complexity for some of these environments by adding another service or application that runs just fine on the same cluster. This level would train the ability to rule out sections of a system that aren’t broken, to identify systems that aren’t interconnected and to work on one system without disturbing the rest of the environment.

    Further environments up the pyramid would have multiple contributing factors, red herrings or complex structures. At this point, some of the environments will need to be hosted outside of the local system. I’m still debating how to do these environments so the tutorial could still be free and easy for new folks to work with–I’m open to suggestions.

    Ideally, there would also be a written guide with answers and tips a student wouldn’t look at until they’ve solved the problem on their own, but they certainly can look up and learn regardless of whether they solved the problem. After all, troubleshooting isn’t a closed-book exam by any stretch of the imagination.

    So how do you start? Find the repo at GitHub. It’s free–open source with an MIT license–and open to contributions. There’s a contribution section of the README on the repo that explains how to get started if you want to send commits. If you want to try using these environments yourself and practice or learn troubleshooting skills, the initial environments are built on Minikube, a tool to run small Kubernetes clusters on your local machine. Follow one of the many excellent tutorials for getting Mikube up and running on your local machine, then follow GitHub’s tutorial on cloning down a repo.

    Alternatively, you can download the repo directly. Once you’ve gotten Minikube up and running and you have a copy of the repo on your local machine, follow the instructions in each environment’s directory to deploy the environment to your local cluster, and start trying to figure out exactly what went wrong.

    Want to help make this project a reality? I would love to have more contributors! In addition to being available online for help, I’m going to be at KubeCon North America 2019 in San Diego. At the conference, I’ll have a Hackers’ Corner set up in LogDNA’s booth for most of the conference, and I’ll be sitting there working on this project, among others, waiting for you. Bring your ideas, your stories, your broken environments, your laptops, your willingness to write docs, your Kubernetes knowledge, yourself—anything that can help enrich this project is welcome.

    To learn more about containerized infrastructure and cloud native technologies, consider coming to KubeCon + CloudNativeCon NA, November 18-21 in San Diego.

    — Laura Santamaria

  • 4 Ways to Give DevOps Teams the Support They Need

    4 Ways to Give DevOps Teams the Support They Need

    If it weren’t hard enough for IT infrastructure teams to innovate and keep the lights on, a host of technologies designed to drive efficiencies are introducing new levels of complexity across the data center. Virtualization, cloud computing, software-defined storage and digital are making it much more difficult for the infrastructure team to control, manage and administer an organization’s IT.

    Just when IT professionals think they’ve figured out the support model, DevOps steps in. As the number of apps continues to explode, more organizations are adopting a DevOps model to accelerate development efforts.  A recent Coleman Parkes Research survey of 1,770 senior business and IT executives revealed 87 percent have adopted DevOps.

    Unsurprisingly, the growing emphasis on DevOps is placing an added burden on infrastructure teams already struggling to cope with the digital transformation of their organizations. Traditional storage systems are often unable to match DevOps’ requirements for an enterprise infrastructure that is simple, flexible and automated. Some DevOps teams also work in silos, which produces an additional level of complexity and opportunities for failure.

    Also, occasional tensions between development and IT operations can flare up over different approaches to infrastructure and competition for resources.

    IT operations teams with traditional storage hardware backgrounds are normally focused on accelerated release cycles, availability, reliability and streamlined data management. Developers, on the other hand, spend little time thinking about what technology they need versus keeping test data up to date with production and having easy access to capacity when needed.

    A Support Structure That Works

    Set against this backdrop, choosing the right platform to underpin these growing changes within the enterprise cloud can play a big part in ensuring a DevOps model that works and makes life easier for the IT infrastructure team. Here are four ways to help make the enterprise cloud infrastructure fit a DevOps model:

    Copy Data Management

    Copy data management provides up-to-date, “virtual” copies of production data to the DevOps team, eliminating the need for physical data duplication. Copy data management helps remove the load on production storage and protects the performance of applications. With snapshots, cloning and replication, modern enterprise cloud systems can enable data copies. The same technologies make it possible to protect and manage the development side of the environment in case of failure or data corruption.

    QoS and Performance Guarantees

    Quality of service (QoS) and guaranteed performance is a significant concern when adopting DevOps. QoS enables IT to allocate different storage workloads to control performance. QoS can set levels of maximum and minimum input/output operations per second (IOPS) or bandwidth. IT can also configure QoS at the virtual machine (VM) or virtual disk (vDisk) level. With the power of all-flash, DevOps can be consolidated on the same platform as production. This makes it easier to access production data sets and reduce the storage footprint.

    Monitoring and Troubleshooting

    The ability to monitor the infrastructure and correct problems and misconfigurations rapidly is valuable when moving to a DevOps environment with continuous integration and continuous deployment. Ideally, the foundations of an enterprise cloud platform should provide basic and advanced monitoring capabilities, predictive analytics, integration with other monitoring tools and VM-level monitoring.

    Automation

    From an infrastructure point of view, it is also critical to provide the capability to provision and manage IT infrastructure programmatically. Automation enables organizations to create and break down environments as needed, incorporate advanced features such as snapshots and cloning as daily workflows, and eliminate the potential for error from manual or interactive configuration.

    As organizations move relentlessly to a mixture of virtual and physical, on-premises and cloud-based IT, infrastructure teams have to manage, control and administer all those resources while matching them to a diverse range of business, development and user requirements. By implementing an enterprise cloud platform that can work across physical and virtual environments, infrastructure teams can help DevOps teams achieve their goals without constraints around performance, scalability, manageability, resilience or flexibility.

    IT organizations are achieving this level of support by replicating the agility of public cloud within the data center to make it easier to manage enterprise and cloud-native applications. This approach can help support the DevOps model and ensure it is implemented with as little friction and disruption to the infrastructure as possible.

    About the Author / Chuck Dubuque

    Chuck Dubuque is the VP, Product Marketing at Tintri. Prior to Tintri, he was at Red Hat where, as director of product marketing for the virtualization business unit, he was responsible for Red Hat Enterprise Virtualization (RHEV) and Red Hat Enterprise Linux OpenStack Platform (RHEL-OSP) market strategy. Before Red Hat, Chuck worked at a value-added reseller for VMware, NetApp and Cisco and also spent several years in biotechnology in marketing and business development. Follow him on Twitter.

  • DevOps Without Scale, Part 2

    DevOps Without Scale, Part 2

    In Part 1 of this series, I covered why it’s important to be aware of your product’s health and how you can do so. I covered the user experience and technical types of metrics and how to get started with collecting them. In this part of the series I will walk you through how to use all these metrics to maximize the user experience and product uptime.

    Metrics, Metrics Everywhere

    There are no silver bullets here and you have to be aware continuously of your product’s health and how it fares against user expectations. It surely takes time and effort, but it’s an essential part of building a good product. Imagine if chair designers didn’t care for how people use their creations; those chairs would not be comfortable to sit in. Similarly, understanding user needs is a paramount part for any software engineer.

    Once you’ve established the basic understanding of your application patterns, you are not done. Every release is an opportunity to introduce more latency or failures and the baseline is ever-shifting. It is easy to significantly affect the latency and as easy to break an edge case and drive user failures up by a few percent with any release. The only way to ensure high-quality user experience is to be aware of your product’s health and your users’ expectations at all times.

    This is the case where less is more and users prefer fewer features that work and are responsive. Try it out and see what your users say!

    User Experience Monitoring

    The first part of a good user experience is low error rates. For web products, these include all http error responses as well as all application errors returned to the user. For APIs and GUIs, this includes application failures (and http errors for REST APIs). One recurring comment I hear is that users submit invalid input, which invariably causes error responses and so it should not count. It is certainly useful to be able to differentiate hard failures from bad user input, but the line is more blurred than you think. For example, a new release might change input expectations and previously valid requests can start erroring out. Is that bad user input or did you just break your product?

    In either case, if the failure numbers are high, users won’t be happy and you need to deal with this problem. The solution here can be non-technical: Maybe you need better documentation or user training, or maybe the API or UX design is cumbersome and drives misunderstanding and misuse. Whatever the case is, it must be addressed. Users are as likely to walk away from a cumbersome product as from a product that doesn’t work.

    As with everything else, I suggest that you identify a few high-level and most important parts of the product and keep track of those metrics. If you do notice an issue, then you have the more detailed metrics on hand for a deep dive.

    The other part of good user experience is performance, and the approach here is no different. Typically you need to track the latency of the same user actions as you’re tracking for failures. Performance tracking is where you really need to understand what’s acceptable and what’s not for your users and that depends on the domain and your users’ expectations. Two seconds to open a reservation on a public travel site seems like a long time, but two seconds to pull up a loan record for an internal user at a financial company is not bad.

    An important fact to remember is that you will always have some failures in the system and that’s normal. No matter how great your product is, how clear its documentation, and how in tune you are with your users, there always will be some failures caused by bad user interactions. At some point, once you feel good about your product overall, you will have to assume that the given error rate is your baseline. Then you need to pay attention to variation over time and revisit critically every once in a while.

    Predictive Monitoring

    In addition to user experience metrics, there are key technical metrics that are good to watch, as they can point to upcoming problems. Some of the common ones are CPU and memory utilization, thread and database pool sizes, VM heap size and garbage collection times. Just like with user metrics, you need to know the operational signature of your application. But in addition, you need to understand how changes in these metrics impact your users, if at all.

    A growing thread pool might mean that response times are slowing down or it could simply mean that more users are accessing your service. A growing email queue size means that emails aren’t going out as fast, but is it bad? Only knowing your domain and users will give you the answer. If the CPU is hovering at 80 percent everything is fine—for now, but you need to scale up quickly before a tiny addition of traffic kicks your whole app over.

    When tracking hardware metrics everyone pays close attention to overutilization because that’s what causes outages. But what about underutilization? Performance fluctuates up and down and at times you can find yourself underutilizing resources. Paying attention to this is a good way to stay frugal and keep costs in check.

    Alerting

    How do you actually know when something breaks? You can’t stare at the graphs all day long. Once you get a feel for the normal characteristics of your system, you’re ready to create automated monitors and alerts. One of the most important things about automated alerting is that it always must be trustworthy. The moment you and your team start getting false alerts, everyone will learn to disregard some of them and the the value of the system will plummet. Thus I recommend you start with auto-alerting only on several most critical metrics. Run the whole system for a few weeks and slowly add more metrics over time.

    Another aspect to be aware of when setting up automated alerting is metric variations over time, the most common one being day vs. night. Nobody wants to get paged because traffic is too low at 3 a.m. At the same time, if your metrics skyrocket or drop suddenly, it can be a cause for concern. Are you being DDoSed or scraped? Is there a network failure that’s preventing your service from being reached?

    Tooling to the rescue, DataDog has built-in capability to create alarms, called Monitors. They provide a few different ways to ensure no false positives. For example, instead of setting absolute thresholds, you can create a change alert that fires only on deltas. And adding a new monitor only takes a few minutes.

    Finally, someone needs to get paged out when an alert goes off. DataDog can do email notifications but nobody checks that at night and even during core hours it takes time to notice a new email. For critical issues like yours, someone needs to get notified immediately.

    Services like PagerDuty and VictorOps have made it trivial to do this. As expected, DataDog integrates with both of them natively. Both services support on-call rotation scheduling with an escalation policy, calling out to phones, text messaging, emails, and more.

    Production Troubleshooting

    So you’ve been alarmed and need to quickly figure out what’s going on and what to do. Your next stop is typically logs, but looking through log files one server at a time is a sure way to waste quite a bit of time.

    Log management tools and services have matured over time and there are good alternatives to Splunk. Loggly and LogEntries cover more than just the basics and provide indexing and search capabilities, graphing of log occurrence, understanding of JSON, and out-of-the-box integration with just about all logging frameworks out there. At their core they provide a quick way to look through all logs without having to go to individual servers. Moreover, you can monitor various aspects of logging à la DataDog and alarm (via VictorOps or PagerDuty) when metrics are off.

    The above approach already solves most needs, but I’d like to touch on a more advanced problem, especially for a microservice architecture. When multiple services are involved, finding the failure at the entry point is easy but figuring out which back-end service failed in the call chain while processing the user request can be quite difficult.

    An architectural pattern that I call log stitching solves this issue. With log stitching you can pull up all logs across your entire architecture for a given user request. This allows you to reconstruct the complete call chain and quickly find the root cause of failures. Conceptually this technique is straightforward: You generate a unique request ID at the entry point, pass it down the chain for every call, and log it. Now you can pull up all logs with that request ID by searching for it. In practice it can get complicated because you need to: a) ensure consistency across all calls, b) account for all entry points including batch jobs, and c) preserve the request ID internally in multithreaded services.

    Log stitching is not something I recommend from the get-go; however, it’s definitely something to consider as your system matures. If you are going down the microservices path, this really becomes a must since tracing call chains get progressively more difficult as the number of services grows.

    Putting it All Together

    There is not much development work in putting it all together, but nevertheless you will need to put in some time. I’d like to emphasize how important it is to work in iterations. Set up one to three alarms, connect your DataDog to VictorOps or PageDuty, study them for a couple weeks and then move on. Keep in mind that you will not get it right at first; your thresholds will be off and you will misinterpret or misunderstand system behavior.

    Operational pattern analysis is not trivial, but don’t spend too much time analyzing the hell out of everything up front. Configure some alarms, get them out there, learn and adjust as you go. When you find an alarm that went off erroneously, take some time to understand why. Specifically, you need to focus on why you thought it was the correct alarm setup vs. why you now know it is not. You will learn by leaps and bounds if you take this systematic approach.

    As I mentioned above, configuring Loggly or LogEntries is super easy and you should do that right away. Log stitching, however, is an involved endeavor that requires planning. Due to its nature it only becomes useful when adopted by all, or most, of your systems. Thus, you should only start it when you feel the need and when you have the buy-in from the rest of your development organization.

    What’s Next?

    Part 3 will cover the topics of continuous integration and continuous delivery, breaking them down into multiple stages of complexity so you can pick and choose what makes the most sense for you from the cost/benefit point of view.