devopsy.kinsta.cloud

Tag: grafana

  • Top Six Open Source Tools for Monitoring Kubernetes and Docker

    Top Six Open Source Tools for Monitoring Kubernetes and Docker

    Kubernetes and Docker are two of the most commonly heard buzzwords in modern DevOps conversations. Docker is a tool that enables you to containerize and run your applications, and Kubernetes provides you with a platform to orchestrate or manage these containers—since managing thousands of containers manually using the Docker CLI would be a practical nightmare. 

    However, simply running thousands of containers and managing them through Kubernetes is not enough. You have to observe and analyze them properly to ensure that your services are up and running optimally. This process is called Site Reliability Engineering (SRE), a term initiated and popularized by Google. Observability and analysis are a major element of SRE. It can be subdivided into the following three areas:

    • Monitoring allows you to extract numerical metrics from your applications and resources that can later be visualized and analyzed to present the current state of your resources. Once the metrics are extracted, they can be used to set up alert rules, facilitate profiling and debugging, and make better SRE decisions.
    • Logging enables developers to debug their containers in case of a failure. Containers are short-lived, and so are logs. Kubernetes and Docker do provide a native way of browsing container logs, but it is very limited in its functionality. As a result, a centralized log plane is a must in any container-oriented environment.
    • Tracing allows you to debug services running on a network and follow a request trail until the source of a problem can be determined. In a microservices architecture, when multiple services/containers send requests to each other to perform one business task, a proper tracing solution is necessary. 

    This article will list and describe the most effective open source tools for monitoring and analyzing services running as containers. 

    Prometheus

    Prometheus is the first tool that comes to mind when discussing open source monitoring solutions. It is very popular in the development community, and it is a graduate of the CNCF project, which focuses on the monitoring side of SRE. Originally created and open sourced by SoundCloud, Prometheus simplifies the process of pulling numerical metrics based on a time series from a given metrics endpoint. It was built to monitor highly dynamic container environments.

    Prometheus is subdivided into three elements: Prometheus server, Alertmanager and exporters. Exporters are stand-alone, independent processes/containers that can be run on your target resource to generate and export metrics via a metrics API. The Prometheus server then takes care of service discovery as well as pulling metrics from the exporters to be stored in the Prometheus DB and later used for visualization or alerting. Alertmanager is responsible for setting up alert rules, analyzing the data in the Prometheus DB and sending alert messages to multiple receivers if a certain rule is triggered. There are a huge number of exporters, listed here, which are both officially supported by Prometheus and maintained by the community. 

    Prometheus has become the industry standard for monitoring cloud-native architectures. While it is known for its simplicity with service discovery, its ease of use, its alerting capacity and its integration with Kubernetes, its polling architecture is not ideal. Currently, the metrics endpoint has to be reachable by the Prometheus server. However, there is a pushgateway implemented in Prometheus that enables metric pushing rather than polling. Another downside of Prometheus is that it doesn’t scale well. This problem can be fixed in the Thanos adaptation of Prometheus.

    Related tools and technologies: Grafana, Cortex, Thanos, Prometheus Exporters, Alert Manager, Istio, Prometheus Operator.

    Figure 1: Prometheus Graphs. Source: SoundCloudDev.

    Grafana

    Grafana is an open source metric analytics and visualization suite. It allows you to create custom dashboards using data taken from multiple sources, such as Prometheus, Elasticsearch, MySQL, Postgres and Redis. Furthermore, Grafana has its own alerting system and a role-based access control (RBAC) system for software. As a data visualization tool, Grafana is famous among Prometheus users for enabling them to effectively visualize the metrics stored in Prometheus. There’s a wide array of official and community-built custom dashboards for various data sources in Grafana, allowing users to easily set up dashboards (found at this link) and get on with monitoring. Grafana offers another associated product called Loki, which aggregates logs in Kubernetes and integrates well with the Grafana UI.

    Related tools and technologies: Loki, Prometheus.

    Figure 2: Kubernetes Resources Dashboard in Grafana. Source: Replex.

    Elastic Stack

    Elastic Stack is a group of open source products from Elastic designed to help users search, analyze, and visualize data from any type of source, in any format, in real time. This product was previously recognized as the ELK stack, where each letter of the acronym stood for one of the company’s main products: Elasticsearch, Logstash and Kibana. Elastic Stack provides both monitoring and logging solutions with the help of its big data database, Elasticsearch. 

    In order to aggregate logs, people tend to use Elasticsearch for storage, Logstash or Fluentd for log streaming, and Kibana for visualization. Fluentd is not part of the Elastic Stack, but it is widely used with Kubernetes to stream logs in place of Logstash, the tool provided by Elastic Stack. Similarly, Metricbeat is used to scrape metrics and set up visualizations on Kibana. The enterprise version of Elastic Stack ships with X-Pack, a set of additional tools that enables reporting, alerting and role-based access control (RBAC), among other features. By default, the Elastic Stack GUI, Kibana, does not support RBAC. You have to use the previously mentioned enterprise version of Elastic Stack to enable it.

    Related tools and technologies: X-pack, Metricbeat, Logstash, Kibana.

    Figure 3: Discover View in Elasticsearch. Source: XpresServers.

    Sensu Go

    Sensu Go is a telemetry and service health checking solution for multi-cloud monitoring at scale. It allows you to gain visibility into servers, containers, services, applications, functions and connected devices across any public or private cloud. Sensu can be run side-by-side with Prometheus to get the best out of both solutions, or it can be run natively without Prometheus. Sensu works best alongside Prometheus, since exporting application level metrics to Prometheus requires loading the Prometheus SDK to your application codebase and exposing a metrics endpoint. The endpoint is then scraped and stored in the Prometheus Server. This may sound like a lot of work—and, sometimes, it is. Sensu avoids this complexity by using a sidecar concept. A Sensu agent is deployed alongside your application. The agent constantly collects and exposes metrics to the Prometheus server without your having to change the application codebase.

    Sensu works without Prometheus too. It can operate natively in Kubernetes, where it has its own server to store and visualize metric data exposed by the previously mentioned Sensu agent.

    Related tools and technologies: Prometheus.

    Figure 4: Events in Sensu. Source: becon.

    Sysdig Inspect

    Sysdig has two open source products: Sysdig Inspect and Falco. Here, we’ll focus on Inspect, which monitors and captures the container processes running in your system and allows you to drill into those processes for post-incident forensics. This enables you to analyze your application performance, troubleshoot errors and monitor any processors that could be misbehaving. Furthermore, if your system is breached, Sysdig allows you to understand how the breach occurred and what data was taken in the process. Sysdig Inspect is a very powerful tool that focuses on the performance tuning and security investigation of your system. 

    Related tools and technologies: Grafana, Sysdig, Sysdig Falco.

    Figure 5: Sysdig Inspect Overview. Source: Sysdig.

    Jaeger

    Jaeger is an end-to-end distributed tracing solution that was open sourced by Uber Engineering. It allows you to monitor and troubleshoot transactions in complex distributed systems. In modern microservice architectures, the majority of operational problems fall under the umbrellas of networking and observability. When a service failure occurs, you have no idea how the requests have gone from service to service over the network to complete a single business transaction. This makes debugging extremely difficult. Currently incubating under CNCF, Jaeger uses tracing to enable root cause analysis, performance and latency optimization, and distributed transaction monitoring. Jaeger works out of the box with Istio, a popular service mesh implementation open sourced by Google. 

    Related tools and technologies: Prometheus, Jaeger, Zipkin, Istio.

    Figure 6: Jaeger Find Traces. Source: Jaeger.

    Conclusion

    These tools are widely used in the tech industry, and they all have their benefits. Most of these solutions, however, require skilled implementation and ongoing manual maintenance that can be a burden for DevOps teams and a distraction from the business. There’s no one solution that can cater to all of your requirements, since each tool focuses on one or two specific aspects of observability and analysis. By mixing these tools together, you can derive a unique solution for your individual business needs. 

    For ease of comparison, the chart below outlines the features offered by each of the tools discussed in this article.

    — Ran Ribenzaft

  • Logz.io Adds Monitoring Service Based on ELK and Grafana

    Logz.io Adds Monitoring Service Based on ELK and Grafana

    Logz.io today at the AWS re:Invent 2019 conference launched Cloud Observability Platform, a monitoring and security cloud service based on ELK—open source Elasticsearch, a search engine; Logstash, a data processing pipeline; and Kibana, a set of visualization tools—and Grafana, an analytics and monitoring platform.

    Company CEO Tomer Levy said it’s become apparent that developers have a strong preference for open source tools for managing cloud-native applications. As such, the adoption of ELK has increased substantially. With the increasing maturity of Grafana, it’s also becoming clear that organizations want to meld the open source analytics and monitoring tool with previous investments in ELK tools, he said.

    However, not many IT organizations have a lot of ELK and Grafana expertise. To fill that gap, Logz.io is launching a managed cloud service based on Logz.io Log Management, an instance of ELK that the company has enhanced with machine learning algorithms and data analytics capabilities to enable DevOps teams to discover and resolve issues faster.

    That core platform is being extended via Logz.io Infrastructure Monitoring, a beta release of a monitoring platform based on Grafana, and a security information and event management (SIEM) platform dubbed Cloud SIEM, based on an ELK stack that includes built-in threat detection tools.

    In addition, Levy said Logz.io plans to add support for Jaeger, an open source distributed tracing tool being developed under the auspices of the Cloud Native Computing Foundation (CNCF).

    Levy said it’s apparent to all that observability is a core tenet of DevOps, which he said is why there are already a half-million developers making use of ELK software. The challenge organizations are finding is that monitoring modern IT environments based on cloud-native applications is cumbersome and time-consuming. Logz.io is addressing that issue by providing an enterprise-grade instance of an open source monitoring platform that comes with hundreds of built-in connectors, which enables DevOps teams to employ a monitoring platform without having to be worried about becoming locked in to a specific vendor because they can switch over to open source code anytime they like, Levy noted.

    Developers tend to favor open source tools because they don’t have to wait on purchasing approval to employ them. However, enterprise IT organizations tend to favor platforms that are supported by a commercial vendor. Logz.io is trying to address that issue by providing support for an open source platform that will make enterprise IT organizations comfortable enough to give up relying on a proprietary monitoring platform. Thus far, Logz.io has more than 700 customers, which the company expects to expand now that it has an instance of its platform that can be consumed as a cloud service.

    It’s too early to say to what degree an open source platform might usurp proprietary platforms in the enterprise. Change tends to come slowly to enterprise IT organizations. However, as organizations embrace next-generation applications based on microservices and containers, enterprise IT organizations are clearly evaluating their monitoring platform options going forward.

    — Mike Vizard

  • The Challenge of Scaling Metrics in Distributed Systems

    The Challenge of Scaling Metrics in Distributed Systems

    Whether migrating from a monolithic architecture or building a distributed system from ground zero, there are many benefits to leverage from a microservices architecture–faster software deployment cycles, enhanced scalability, improved isolation of risk. However, distributed systems are complex and as such, accompanied by many challenges, particularly when it comes to the scalability of metrics.

    The Importance of Observability in Distributed Systems

    In order to realize the benefits of a distributed system, a strategy of data-driven optimization is essential. The paradox here is visibility simplifies complexity, while the complexity of visibility is dramatically increased in distributed systems.

    As New Relic points out in their eBook Cloud Native Is the New Normal, traditional monitoring approaches need to be adapted to meet the demands of decentralized architectures, including:

    • Overcoming the loss of end-to-end monitoring capabilities by learning how to connect the dots across a complex network of environments.
    • Developing tactics to quickly drill down into issues across interconnected parts to reduce mean time to resolution (MTTR).
    • Introducing proactive reliability strategies that focus on measurement first, followed by improvement.

    Finding the Right Balance Between Monitoring Nothing and Monitoring Everything

    Most application engineers leverage some sort of metrics tooling such as Grafana, Prometheus or Graphite to track vitals for their system health. While monitoring is essential, there is such a thing as bloat when it comes to metrics ingestion; and this overload translates to scalability challenges that affect performance, cost and reliability.

    So, how do you know what to monitor? In order to derive the most valuable insights from your system, you must first understand and identify the components, and more specifically which attributes within those components drive key metrics. Knowing this requires an intimate familiarity with how all the pieces work together in your distributed system.

    Google’s Site Reliability Engineering book provides a framework for determining what to monitor in distributed systems. They break signals down into four different categories:

    • Latency: Understand resource requirements and identify bottlenecks.
    • Traffic: Monitor system response to varying levels of load; most useful when paired with other indicators.
    • Errors: Track frequency and nature of interruptions in request/response flow.
    • Saturation: Surface capacity-related issues.

    Even when you’ve identified the most important metrics to track, scalability is still a challenge that must be addressed, particularly when dealing with applications that handle massive volumes of data across a complex application architecture.

    M3: Uber’s Open Source Large-Scale Metrics Platform

    A solution directly addressing the challenges of distributed metrics at scale is M3, an open source project developed by Uber’s engineering team.

    Before using M3, the Uber team sent metrics to a Graphite stack with storage handled using the Whisper file format in a shared Carbon cluster. From there, they created Grafana dashboards to visualize the data and used Nagios for alerting based on Graphite threshold checks via source-controlled scripts. As the volume of metrics continued to expand, manual processes to scale the solution became unsustainable and repeat failures resulting in permanent data loss led them to build a more fault-tolerant, scalable system.

    M3 is focused on providing a reliable storage solution for large-scale metrics over long retention periods, solving inherent single node-associated limitations with Prometheus by offering a multi-tenant store for Prometheus metrics. A key component of this solution is M3DB, the scalable storage backend for M3.

    M3DB, built in Go, is a distributed time series store and reverse index with configurable out-of-order writes. By enabling the storage of real-time metrics over long periods of retention, M3DB can replace a Prometheus cluster. In developing the M3DB solution, the Uber team essentially deconstructed Prometheus into microservices to enable the scaling of each component individually. By taking this approach, M3DB is able to offer better reliability, efficiency and performance.

    At a minimum, M3DB can be leveraged as remote storage for Prometheus (or used to replace any other remote storage solution). There are, however, other features the solution offers, including an external aggregation engine and a separate query engine.

    The M3DB project is still under development with a growing community of contributors.

    To learn more about containerized infrastructure and cloud native technologies, consider coming to KubeCon + CloudNativeCon NA, November 18-21 in San Diego.

    — Molly Wojcik

  • Challenges of Big Infrastructure Monitoring

    Challenges of Big Infrastructure Monitoring

    The growing complexity of today’s IT reality is indisputable: Software is becoming more complex, and the infrastructure it runs on is equally so. This complexity comes, in part, from adding layers upon layers to the infrastructure we manage—from bare metal servers to VMs to containers to function-based computing—and from how quickly we cycle in new technologies and how slowly old technologies are cycled out.

    Ephemeral and multi-generational infrastructure (often side by side) are the new normal. Old and new technologies co-mingle, and while new technologies may be shiny and exciting, they quickly become ordinary as the next new thing comes along.

    Here’s a sobering stat: In Q4 2017, IBM’s mainframe business grew by 32%. So even while cloud, AI and security initiatives are on the rise at IBM, mainframes sustain a bulk of their business. Old tech perseveres, and we have to find new solutions to bridge the gap between old and new.

    The hodgepodge of technologies—and the challenges they bring—are compounded further when we’re talking about big infrastructure.

    What Qualifies as Big Infrastructure?

    For the purpose of this discussion, big infra is an organization that has been around for at least 10 years. That means they’ve had success, they’ve grown and their infrastructure has grown along with them, and the technologies at play have expanded as well.

    Your organization’s exact age, size and revenue aren’t particularly material; what matters is you’ve had enough time to accumulate resources, people, practices and technologies to create what I think of as a trail mix problem—basically enough legacy infrastructure to start creating problems.

    The Challenges of Big Infrastructure

    The challenges of big infra stem from the technology itself and from the way people use that technology. As an organization grows, its teams often become larger and more siloed, and departmental goals become disconnected. Even when working toward the same goal, departments diverge in their methods for reaching that goal.

    Now, add tools and technology to the mix: Over time, teams choose tools they prefer, or build their own, which works out okay for a while until they have to collaborate with another department that uses different tools and a completely different set of practices.

    Now, add in different work styles and personality types and company politics, and everything gets tangled up fairly quickly. Trying to keep track of who’s using what, how they’re using it and what’s working (or not) is no easy task.

    One customer, for example, had one team using Chef and was all in, and were trying to get buy in from other teams as well, but there was another team that was early days Docker, hand rolling Docker hosts, and didn’t want to adopt Chef. Then there was a different team who was early days Kubernetes and didn’t want to use Docker, and yet another team using Puppet. Once they had to work with each other, things broke down—someone has to win out. You almost have to maintain a logical map of what team uses what tools as a precursor to prepare for a meeting with these business units. Not only is there no consistency in how applications are deployed, there’s also no consistency in how these different technologies are monitored.

    For example, in the past, Netflix stated monitoring accounts for 30% of their infrastructure cost, and recently mentioned they’re planning to spend $1.9 billion on infrastructure in 2019. Think about the scale and complexity involved—they’re storing and analyzing the logs, behavior and metrics for systems supporting some 125 million subscribers.

    Connecting Disparate Data Across Big Infra

    So, how can you streamline and connect all of the data spread across fragmented tools and systems?

    One solution is a data lake, a centralized repository that lets you funnel data into one place, so you have a holistic view of your infrastructure. In reality, big infra ends up spawning multiple data lakes that become siloed from one another.

    You also likely have a square peg/round hole situation—data in one format needs to be transformed to another format before it can get into one of the data lakes.

    This is where a monitoring tool becomes essential. Look for a monitoring tool designed to collect data in disparate data types and transform it into any number of formats, so you can set up an automated workflow to get data flowing from legacy systems (e.g., mainframes) alongside modern infrastructure like containers.

    An example of this idea is with Sensu, and our support of the Nagios plugin spec and Prometheus exporter metrics (amongst many other metric formats). By using a monitoring tool that works with various metric formats, companies can plug the service checks they’re already using plus data from multi-cloud infrastructure into their monitoring setup and get valuable information about their systems.

    A visualization tool such as Grafana is the icing on the cake. With Grafana, you can hook up data wherever it’s stored (e.g., Graphite, InfluxDB, Prometheus, etc.), visualize the data and display it on shareable graphs and dashboards. Using a monitoring tool as a data source for Grafana means companies can display monitoring incidents and inventory—and even mix that data with different data sources on the same Grafana dashboard. Combining a holistic monitoring solution with Grafana’s visualization capabilities gives a data-driven view of your infrastructure that you need—even if it remains a multi-generational, multi-siloed, multi-data-laked, 2,000-different-format-typed trail mix of big infra.

    — Sean Porter

  • NS1 PARTNERS WITH RAINTANK, ADDS GRAFANA TO GROWING INTEGRATION TOOLSET

    SANTA CLARA, Calif. and NEW YORK – June 21, 2016 – (O’Reilly Velocity, Booth #511) – NS1®, the leading provider of intelligent DNS and traffic management services, announced today it has partnered with raintank, the company behind Grafana. Thousands of enterprises use Grafana to gain single-console visual analytics of their IT infrastructure. With the new application for NS1, Grafana users can now add key DNS analytics into their IT management dashboards. The Grafana application for NS1 is available now on Grafana.net.

    Click to Tweet

    • Aggregating metrics and performance: Grafana, the de facto open-source solution for visualizing time series data, consolidates data from the entire stack into a single platform. Features include pluggable panels and data sources for extensibility, including fully featured dashboards with rich visualization options.
    • Visualizing NS1 data in Grafana: Provides teams with a real-time view into DNS traffic and infrastructure health. The NS1 app provides real-time and historic visibility into query-per-second (QPS) traffic and NS1’s high-frequency monitoring.
    • Simple and elegant implementation: This integration provides a seamless way to obtain visibility into NS1 metrics and usage. The App is configured in mere seconds using Grafana’s plug-in platform and allows users to simply enter credentials to automatically generate dashboards from their NS1 data.

    Jose Hernandez, security operations, Zenedge, said: “NS1’s Grafana app helps us aggregate our DNS performance along with other critical metrics into a single dashboard. Configuration couldn’t be easier and gives us instant visibility into our NS1 account usage and monitoring tools.”

    Raj Dutt, CEO, raintank, said: “We’re excited to partner with NS1 to help streamline data visibility for our mutual customers. The powerful combination of NS1’s next-gen DNS platform and Grafana’s elegant visualization framework is perfect for networking and DevOps experts.”

    Alex Vayl, cofounder and VP business development, NS1, said: “Our goal is to provide developers with the tools they need to help them succeed throughout the entire DevOps process. By adding Grafana to our toolkit, we now are able to provide our customers with the visual analytics they need to deliver real-time, actionable business insights.”

    Additional Resources:

    About NS1

    NS1’s intelligent DNS and traffic management platform, with its data driven architecture and unique Filter Chain routing engine, is purpose-built for the most demanding, mission-critical applications on the Internet. NS1’s comprehensive platform technology leverages infrastructure, application, and network data to make intelligent routing decisions in real time, ensuring optimal application performance and reliability. NS1’s customers include leading brands like, Yelp, Imgur, MaxCDN, Collective Media, OpenX and many more.