devopsy.kinsta.cloud

Tag: log management

  • Log Management: What DevOps Teams Need to Know

    Log Management: What DevOps Teams Need to Know

    One of the most important shifts of this decade is the rise of interconnected data across distributed systems. This explosive growth not only sparked the need to rethink cloud and global IT strategies, but it also disrupted traditional development, DevOps and ITOps practices. These changes are pushing requirements for higher-velocity development of features and products—together with more responsive support requests—onto technology professionals. These new requirements have prompted teams to evaluate new tools and agile processes that help support the needs of a competitive modern business.

    Regardless of the pace of change within your group, basic functions such as infrastructure monitoring are crucial to operating production applications. At this new speed of business, concepts such as observability are important for enabling successful deployments with fewer user interruptions—not only in production, but also in the early stages of your continuous integration/continuous delivery (CI/CD) pipeline. As part of “shifting left,” log and event management in your CI/CD pipeline empowers developers to monitor and observe application behavior before releasing it to production. While some might say this adds additional layers of process, time, effort and due diligence in dev environments, it can help you evolve your organization’s fundamental software development behaviors while reducing preventable issues from post-deployment into production. It can create a smoother, more seamless user experience and reduce the need to fight fires or rearchitect your solution after the application is live in production and at scale.

    Setting the Testing and Dev Stage

    The rapidly evolving technology landscape has increased the need for log management and observability across distributed systems and containers. Changes to how applications and services are created, the ability to deploy applications across combinations of global logging-as-a-service (LaaS) providers and the pervasiveness and volatile lifespan of containers, along with the capacity to build services using various development languages, has increased the need to collect, monitor and trace data points across connected systems, which provide critical end user-facing value.

    Traditionally, many considered log management a tedious task because each instance of a new server required you to run a search command across the logs local to that server. That approach isn’t scalable when every cluster of more than 20 servers is built up in minutes, and each application has independent logging formats. In this scenario, you’d need to run 20 individual searches across multiple systems, compare timestamps across servers residing in different time zones and so forth. With the decreasing cost and increasing prevalence of containers and virtual systems, the number of systems running within a typical organization has grown exponentially as the business prospers.

    In addition to increased usage of virtual systems and containers, independent scrum teams of developers can build in parallel using technologies which are the best fit for the services they are producing, while also sharing environments for builds, staging and production. With these distributed services, aggregated log systems can quickly determine the root cause for debugging their application’s errors, exceptions or performance bottlenecks. A log aggregation system can reduce the element of “personalization” that can often happen when developers are using different technologies with various logging formats, and also reduces the noise by filtering out information that delays the team’s ability to find the source issue. This can cause confusion or slowed productivity when one developer is left to understand another’s non-standardized, yet shared, workload.

    The reduced cost, increased collaboration and prevalence of container services are collectively driving an uptick in virtual environments and containers, often requiring more logs to be created in testing and dev environments, which can force organizational shifts to enable full observability across their infrastructure.

    The 3 Pillars of Observability

    Businesses are in the midst of a shift toward adopting these principles—the three pillars of observability—and using a three-pronged method of monitoring that takes data from several vantage points to depict a more accurate and consumable view of overall health and stability:

    1. External Monitoring: An example of this is health checks run against your internal and external applications and websites to see the “digital experience” of your users.
    2. Metrics and Distributed Tracing: This enables you to trace communications between applications distributed across systems/containers to identify errors and exceptions from your applications and resolve latency quickly.
    3. Events and Logs: Data, which helps provide contextual information about events, enables you to identify issues in the code when combined with information from the first two pillars.

    The above types of information are critical to DevOps to provide elastic and resilient services. Bringing application, system and infrastructure events into logs together with external monitoring and metric collection is quickly becoming the standard. With powerful log-parsing features such as live-tail search accessible from a web browser and the ability to visually pair log insights with added health metrics, the real value of log data is made apparent with little effort.

    Site reliability engineering (SRE) is another area where significant evolution has occurred. ITOps has always been asked to react to fire and run from fire to fire. The evolution from ITOps to DevOps to SRE is forcing the alignment of different types of data to be used very early in a CI/CD pipeline, and enabling SRE teams to get involved in architectural discussion at project inception instead of in the latter stages or at release. The SRE is more involved in monitoring, understanding the development and architectural principles used to develop application services and build in automation, and high availability and circuit breaker patterns from inception.

    This approach has allowed teams to be proactive and responsive rather than reactive. It has equipped our teams (which keep our business running successfully and our customers loving what we do) with the ability to focus on being scalable, elastic and resilient. Instead of troubleshooting systems in production, we auto-scale and replace to minimize disruptions and investigate that incident offline when possible.

    Best Practices

    Given the increasing pace of application development required to support the needs of the business, the spread of containerized applications and virtual environments and the emergence of new disciplines for observability (due to the lowered costs and increased reliability), log and event management has become a critical aspect for all involved in building, supporting and even using mission-critical applications. It’s moving to a defined area where logs should be compartmentalized, and access control should be defined:

    1. Compartmentalizing Logs: Critical as a best practice so we can differentiate development, staging and production environments and segregate logs based on practical grouping. Using a log aggregation service provides you with a consistent experience and a common set of capabilities across all your environments.
    2. Access Control: Another best practice that should be implemented more broadly; for example, developers don’t need access to all of your production logs if approximately 95 percent of their work happens prior to reaching production environment. In other words, even though tech pros are working to “de-silo” log management, we still want to silo access control and manage visibility. This can also help to ensure personally identifiable data (PII) isn’t exposed to anyone it shouldn’t be.

    Looking Toward the Future of Log Management

    As the landscape evolves and best practices become more defined, we should expect revitalized strategies with elevated expectations for the traditional scenarios where basic/free monitoring is leveraged. While the pillars of observability are leveraged more broadly in the ITOps and DevOps communities, we’ll be able to move more efficiently from an end user-facing incident—for example, an application error traced across distributed systems—straight into the logs to find the specific error causing the issue. We’ll be able to see greater correlation between logs and other custom metrics, as well as much broader use of these techniques outside of production environments.

    DevOps teams need an intuitive event and log management system that is as de-siloed as we are. No longer can log management systems be relegated just to one on-premises storage system or one virtual environment; they need to fit the reality of how work is increasingly done today and will be in the future.

    — Keith Kuchler

  • We Still Haven’t Solved the Logging Problem

    We Still Haven’t Solved the Logging Problem

    At a company I previously worked at, I helped build out the initial infrastructure to support a large cloud security product. It was one of the first times I had the opportunity to build something from nearly the very beginning. Like many operations professionals, so much of my career prior to this experience had been spent cleaning up other people’s messes and technical debt that I was finally excited to “do things right” from the onset.

    Looking back, I wish I knew that building our monitoring platform would be hampered without the ability to retain and analyze both warm and cold log data. Here’s why.

    Time Series to the DevOps Rescue 

    Hindsight is almost always 20/20, and today I can proudly say that my team and I built a killer app monitoring platform. My goal was to make it easy for Dev and Ops folks to report time series metrics for their applications and build dashboards and alerts for app health. And we did just that.

    While many ops colleagues would say that monitoring journeys usually start by capturing application logs with open source tools such as Logstash/Elasticsearch, Graylog2, or even buying commercial services from Splunk or Sumo Logic, not all journeys start that way. My team and I took a different approach and optimized for time series metrics versus the more traditional structured log management approach, leveraging newer toolsets such as Librato for real-time operations monitoring and performance analytics. Librato enabled us to scale much faster than with traditional log solutions and gave us almost instant visibility across our containers, cloud servers and the apps themselves.

    Logging is Too Expensive 

    Using hosted services such as Librato to manage our time series metrics worked fine for a while. However, the cost of SaaS monitoring eventually grew, and, since we had the time and energy to bring metrics in house, we deployed Graphite, the open source enterprise-grade monitoring tool. Graphite allowed us to continue enabling our engineers with affordable access to granular time series metrics to help them manage their applications.

    While this allowed the business to respond quickly when an issue arose, time series metrics were only part of the solution. They allowed us to see what was happening, but not always why it happened. While time series metrics and even some of the more advanced tracing technologies including Jaeger and Opentracing are fantastic to help debug complex distributed systems, they fall short when investigating and analyzing causes for problems, issues and hacks.

    Digging Into Your Log History Gets Expensive, Quick

    Tools such as the ELK stack have been booming, with Elastic alone hitting more than 100 million downloads last year, which shows that DevOps teams want more from traditional log management solutions. They’re looking for simplicity, ease of use and affordability, among others. With JSON effectively the de facto standard for logging, most teams take log data with its predefined schema and then index it into Elasticsearch/Lucene. Unfortunately, the Lucene indexing mechanism results in an unintended consequence: a 5 to 10X increase in disk usage. And just like that, ops cloud budgets get blown away when they realize that 10GB of JSON data indexed into Elasticsearch could be 50GB of disk used. You wanted that index to be replicated for disaster recovery as well? Then add 100GB of storage for every 10GB of logs.

    DevOps Normal: Still the Tough Choice Between Retention and Cost

    Unfortunately, most people choose to simply purge data from our “hot” Elasticsearch clusters as soon as we possibly can to minimize cost issues. This is far from ideal.

    Companies moving to microservices-based architectures, deployed on containers such as Kubernetes, are going to be in for a rude awakening when they start building out their centralized logging platforms to keep up with the volume. Security teams are going to feel the burden as well, as they come under increasing pressure to keep data around for long periods of time due to compliance and audit reasons. When the next big web-facing vulnerability comes out, the ability to go back weeks or months will be key to see what IP address accessed a potentially malicious endpoint. With that data available, we could then take all those IPs and correlate them with other access logs.

    Is centralized logging any better than it was 10 years ago? Or has all this data generated exploded so much that the tools we have today are just barely keeping up? One thing is certain, this problem is far from solved, and many companies may find themselves drowning in their own data lakes.

    — Pete Cheslock

  • Sauce Labs Streamlines DevOps with Sumo Logic

    Sauce Labs Streamlines DevOps with Sumo Logic

    Log management and analysis is a crucial element of effective DevOps. In an effort to work more effectively and efficiently at the same time, gathering and analyzing feedback provides essential input that organizations can use to correct issues and enhance performance. There are a variety of log management tools out there, but Sauce Labs solved its log management and analysis challenge with Sumo Logic.

    devopsy.kinsta.cloud recently had a chance to sit down with Joe Alfaro, VP of Products for Sauce Labs, to talk about the decision to go with Sumo Logic’s software-as-a-service (SaaS)-based log management solution.

    Prior to making the switch to Sumo Logic, Sauce Labs was using an ELK stack for log management. Alfaro explained that the ELK stack was tedious to maintain. “Our ELK-based solution required constant care and feeding. We were always having to make sure we had enough disk space and we were always fixing bugs or adding new features. We had a small team dedicated to management of this solution. In addition, it was difficult to bring developers up to speed on the tool, since documentation was not always kept up to date and there were no training materials.”

    The effort required to troubleshoot and maintain the tool itself got in the way of Sauce Labs’ ability to use the tool to debug production issues, and used precious developer resources that were needed elsewhere. Sauce Labs decided it needed a new tool—something that was very flexible and cloud-based, so the company wouldn’t need to worry about maintenance or storage.

    To find this new tool, Sauce Labs did proof-of-concept testing with a handful of well-known vendors: Graylog, Loggly, Splunk and Sumo Logic. Alfaro was emphatic about the Sumo Logic experience. “Implementation was ridiculously easy. We were up and running within a day,” he said, adding, “Sumo won out as it has the combination of: ease of use, includes all of the features we wanted, is cloud-based and within our budget.”

    The switch to Sumo Logic has also helped the company streamline its DevOps practices. Sumo Logic provides a communication tool between the developers and ops engineers, acting as a common language that allows DevOps teams to work together to identify the root of problems quickly and easily.

    Alfaro shared that the bridge between Dev and Ops provided by Sumo Logic has led to improved service, as well as savings in both time and cost. “It also finds patterns in our logs that lead to better analysis in discovering problems. This machine-learning-like pattern discovery is invaluable to getting to the root is issues. These days we spend minutes finding problems rather that the hours or days we used to spend in the past, before Sumo. This leads to more uptime and more developer productivity.”

    Of course, your mileage may vary. Graylog, Splunk and other log management and analysis tools are formidable in their own right. If you’re in the market for a new log management solution, though, Sumo Logic should be on your short list for consideration. If your results are anything like Sauce Labs’ results, it would be a crazy move not to check out Sumo Logic.

    — Tony Bradley

  • 7 things 2016 has in store for the future of DevOps and IT

    7 things 2016 has in store for the future of DevOps and IT

    With only a couple weeks left in 2015 it’s time to start looking ahead to what next year has in store. Let’s be honest—most of us have already mentally checked out for the holidays anyway, so it’s already 2016 for all intents and purposes. I spoke with a handful of industry experts to get some thoughts on what we have to look forward to in 2016.

    Let’s just jump right in:

    1. Log Management will play a larger role

    According to Christian Beedgen, CTO of Sumo Logic, log management will emerge as the most underrated opportunity for IT. “Thinking about log data may seem too far in the weeds for most tech industry professionals. However, using analytics to monitor, manage and gather insights from logs will be the only near-perfect way to make sense of the increasingly complex and cloud-based architectures.” Log management isn’t exactly new or cutting edge, but Beedgen believes that 2016 will see more vendors trying to move into the log management space, and more organizations trying to use log management to make sense of their big data mess.

    1. Business leaders will demand better insight

    On a similar note, Rick Fitz, senior VP or IT Markets for Splunk, believes that analyzing data and metrics will also play a role from a business strategy perspective. “In 2016 we will see data science demystified, moving from Ph.D. to MBA, with easier ways emerging to consume, analyze and correlate machine data. As business leaders increasingly go digital, this will result in greater “digital insight” into engagement, satisfaction, competition, effectiveness, response, revenue and other business goals.”

    1. Sharing economy will transform IT

    Companies like Uber and AirBnB have transformed entire market segments. The sharing economy has changed the way businesses engage customers and deliver services, and that includes IT through aspects such as cloud services, microservices, APIs and more. Fitz predicts, “Next year this ‘Uberfication of IT’ will turn into a ‘Balkanization of IT,’ driving stakeholders to demand better insight, governance and control of federated technologies to integrate adoption, usage, monitoring, security, cost control and more across shared services.”

    1. Enterprise DevOps will grow rapidly

    The DevOps revolution has taken IT and app development by storm in recent years, but most of the traction has been in startups or smaller businesses that have more freedom to shift gears and embrace new concepts. Enterprise adoption has been growing, but at a much slower rate. Andi Mann, chief technology advocate for Splunk, said, “In 2016 we will see rapid acceleration of DevOps with 50 percent or more of large businesses finally adopting ‘Enterprise DevOps’ — a pragmatic, scalable approach to adopting the culture, process and technology changes at the heart of DevOps.”

    1. The singularity approaches

    OK. That may be an ominous over-statement. In a nutshell, Sumo Logic’s Beedgen claims that the line between machines and humans will continue to fade. The foundation of this is automation—leveraging technology to perform mundane or routine tasks and sift through mountains of data exponentially faster than a human can. As artificial intelligence evolves, though, the scope of what can be automated expands as well. “In automating processes, a human can today wield the power of a legion of operators with a single hand,” stated Beedgen. “Likewise, machine learning will increasingly be adopted for its usefulness in augmenting human understanding of complex interaction and large data sets by uncovering the unknown unknowns.”

    1. Automation and analytics will improve IT service health

    DevOps and microservices have transformed many areas of development and IT infrastructure, but many organizations still rely on decades-old monitoring tools, arcane metrics, and specialized admins dedicated to managing the cumbersome process. Splunk’s Fitz thinks that is getting ready to change. “In 2016, IT analytics will be liberated as easy-to-use dashboards will correlate all sources of service health—legacy tools, APIs, mobile apps, Web logs, wire data, containers and more—while predictive analytics will automatically respond to prevent quality issues.”

    1. DevOps is here to stay

    From the point of view of DevOps practitioners or companies that have adopted DevOps tools and practices, this may seem both painfully obvious, and yet still be good news as well. Christian Beedgen predicts not just that DevOps is here to stay, but that the importance of DevOps and the sphere of influence affected by DevOps will expand. “Increasingly, DevOps will permeate all manner of businesses that rely on information technology to compete. Bimodal IT (the practice of managing two separate, coherent modes of IT delivery) will become the equivalent to “private cloud”—the last forced resistance of the old guard. Both will be forgotten in a short number of years, as public cloud and lean and agile processes dominate the businesses that still compete.”

    There you have it. Now you can enjoy your holiday break in peace and come back in 2016 with an idea of what you have to look forward to in the year ahead.