Elastic and Grafana Labs have allied to create a plugin through which DevOps teams that are employing Elasticsearch within a DevOps workflow will be able to invoke a monitoring platform, based on Prometheus, that is managed by Grafana Labs.
Raj Dutt, Grafana Labs CEO, said both companies are dedicating engineering resources to building and maintaining a plug-in for a service, based on open source Prometheus software, that is rapidly becoming a de facto standard for observability in DevOps environments. One of the primary reasons for that expanded usage is that Grafana Labs doesn’t require DevOps teams to move data into a central repository to analyze it, Dutt said.
The overall goal is to eliminate the “tunnel vision” that results when relying on tools that only visualizes a narrow set of metrics, Dutt added.
The cloud platform provided by Grafana Labs is based on open source Grafana visualization tools. The primary tool for visualizing Elasticsearch data is based on open source Kibana tools. Grafana, however, is based on a fork of Kibana, so the plug-in represents an effort to bring those two open source communities closer together. The two companies plan to collaborate on including more aggregations, broader query language support and support for space-saving constructs like rollups as the partnership evolves.
Grafana Labs provides a cloud service that aggregates data, pulled from multiple instances of Prometheus, deployed on clusters and servers. Prometheus, especially, gained a lot of traction as organizations began to deploy Kubernetes clusters that are also based on open source software. Both Kubernetes and Prometheus are being developed under the auspices of the Cloud Native Computing Foundation (CNCF).
Once DevOps team initially employ Prometheus to monitor Kubernetes, they frequently expand deployments to monitor additional platforms. That shift has resulted in many providers of proprietary monitoring tools to extend the data they consume to include the metrics captured locally by Prometheus. Grafana Labs claims it has more 1,000 customers, including Bloomberg, JP Morgan Chase, eBay, PayPal and Sony, for a total of more 650,000 active installations of Grafana around the globe. Existing customers of Elastic that are paying for Elasticsearch support include Cisco, eBay, Goldman Sachs, Microsoft, The Mayo Clinic, NASA, The New York Times, Wikipedia and Verizon.
It’s not clear to what degree Grafana and Kibana might ultimately re-converge one day, but it’s clear there will be a fair amount of ongoing cross-pollination across the two projects. In the meantime, DevOps teams should be able to opt for whichever open source visualization framework they prefer alongside Elasticsearch.
In the longer term, it’s clear greater appreciation for observability platforms that provide more context will drive a wave of consolidation among all the monitoring tools that enterprise IT teams rely on today. As part of the transition, enterprise IT teams might even standardize on a set of visualization tools that would eliminate the need to master different user interfaces. After all, the only way to actually achieve observability is through a set of tools that makes surfacing the context required easier to consume.
With the advent of DevOps, we are now moving toward more collaborative workflow models, where the integration of development and operations teams make it easier to quickly move development projects into production, and with fewer roadblocks. However, within DevOps, many of our activities are not as clear-cut as they were in previous siloed methods. While previous methods made it easier to keep track of activities and processes, these systems were not particularly efficient.
In the DevOps model, moving software from development to production is much faster. But, we need to be able to identify actions that occur in our increasingly complicated applications. We now have many chefs working at the same time, and visibility suffers.
Logging helps us better understand applications throughout each stage in the process so that we can identify problems or larger issues that might arise already during the development process. This is particularly true within CI/CD environments where items are regularly pushed into production. We need methods for identifying not only if there are critical errors, but also the ability to attribute the errors to specific versions throughout the cycle.
To deal with the increasing complexity and decreasing visibility, DevOps identifies complex issues using logs to keep track of services, APIs, containers, infrastructure issues, network activity and also security related logs.
Log files provide a tremendous amount of useful information about what goes on beyond the CI/CD pipeline and also identify issues that directly impact customers. Good log analysis tools make it possible for DevOps teams to get much closer to the end user. And open source is the preferred starting point for many organizations.
Why Open Source?
Lack of Lock-in
Many DevOps teams make use of a wide-range of tools and find that they can work much better without vendor lock-in, which can be caused when using proprietary stacks exclusively. This makes it far easier to make any needed changes as the need arises, and not be reliant on individual vendors. This also helps avoid many risks, such as vendors disappearing, ending support, changing pricing models, etc.
Widely Used
Within CI/CD, there are many commonly used open source tools, which makes it easier for new members to be on-boarded. There is already a widely accepted suite of tools that almost all DevOps teams use. It’s hard to imagine working without Jenkins, Docker,Kubernetes, Git, etc.
Regular Skill Development
Open source work ensures that DevOps teams are actively learning and directly interacting with software and systems. It becomes a way of ensuring that skills and abilities of teams are continually staying updated. This can translate into better troubleshooting and problem solving skills. When working with closed systems, there’s considerably less opportunity for knowledge advancement, as closed systems tend to encourage rote work with existing software rather than skill development.
Quality
Fears about subpar open source software are largely unfounded,particularly if the projects are widely used and have a large community. There are far more developers working on a project in the open source community than in proprietary systems. Closed systems can lag behind, and there may actually be bad code but that is hidden due to its closed-source nature. The ability to make modifications to create functionality beyond that which already exists in closed source systems, makes open source extremely appealing for many DevOps personnel.
Some Drawbacks
It’s important to note there are a few drawbacks when attempting to work entirely within open source environments. Many resources used are not open by nature–for exampleleadingclouds are decidedly not open source in nature. Individual users (outside these companies) have much less control over how they actually operate. However, if we look at the toolkits that we described above, we can think of open source tools as needed resources for working in and around these environments.
Another drawback to consider is the actual costs of managing open source software versus a fully-managed service such as Coralogix, for example. Many times, teams will start with the self-managed approach and move to a managed service when faced with scaling and complexity issues as the organization grows.
Types of Features to Look for in Open Source Logging Models
When setting up to work with open source logging tools, it’s important to understand there is arange of different types of models available, which can work better or worse depending upon needs and capacities.
OpenAPI: provides a useful interface into many different systems. It is language agnostic and can generate clear documentation of methods, parameters and models. In most cases it will work within RESTful interfaces, but will also work with other protocols such as SOAP.
Open Standards: Even if a tool is not open source, typically these should follow a set of standards to make sure that different pieces can talk to each other. OpenAPI is one example, but not the only one.
Federated Model: This is a broader type of open source model which is like many default open tools, but allows a much wider set of input and may allow individual users to provide input and maintain some control over local development areas. This means that aggregation, processing and control remain local. However there’s still a central organization which will be able to collect summarized or complete code. The advantage of a federated model is it increases flexibility for individual teams, while still contributing largely to the project as a whole.
Elasticsearch, Logstash, Kibana (ELK)
One of the most popular open source logging stacks is known as ELK. It is particularly powerful and effective for the purposes of those wishing to create a clear aggregation and understanding of their log files, with the ability to find both problems and solutions quickly.
ELK is a combination of three tools.
Elasticsearch: a popular NoSQL Lucene search engine.
Logstash: a pipeline system which will take data from logs, ingest it and transform it into a data store (such as Elasticsearch), so that it can be searched and analyzed.
Kibana: a tool which makes it possible to create clear visualizations for Elasticsearch to appear in a human-readable format.
What ELK provides is a single place for all of your data, where it can be stored, searched and analyzed in a method. It is useful for DevOps teams wishing to make use of the vast amounts of data stored in log files.
Upfront expenses for open source tools like ELK are minimal; the software itself is free. However, this does come with a caveat. Professional management of ELK tools is required, otherwise costs can grow quickly out of control.
Generally, for low volumes of data, one can manage ELK for as little as a few hundred dollars a month on a platform such as AWS. However, without careful management, the cost can grow exponentially for large amounts of data.
Work required for ELK involves:
Setup, which may take some time but that’s typically not a problem for many ops teams—and is a project many will enjoy.
Maintenance of elastic search clusters can be quite a bit of work—several hours per week simply solving issues or resolving downtime. This can vary based on the size and growth of clusters. The more activity you have occurring, the more time you will need to spend maintaining your logs.
FluentD
FluentD is an open source alternative to Logstash which can collect, parse and transform data for further analysis. It has features which may be particularly helpful if you are running your logs in a Kubernetes environment.
One of the drawbacks of Logstash is that it is written in JRuby, which means it requires a Java runtime for each implementation. If you are running many different microservices in Kubernetes, this can result in a large amount of your memory being taken up.
Because FluentD is developed in CRuby, it doesn’t require as much memory to use with each new pod creation. This can help solve some memory allocation issues which could cause your logs to slow down your applications.
FluentD also has a wide range of plugins, which can cover pretty much any use case that you come up with. Another advantage is that its tag-based routing makes it slightly easier for developers to work with than Logstash.
What to Log?
With continuous integration, there are many things that merit monitoring. These can include everything from checking the frequency of merge conflicts in Git, or whether pulls are preceded by pushes prior to any other pulls.
Are you receiving a high number of compiler warnings? You can identify code that has a strong likelihood of breaking builds, change your unit testing procedures and implement better automated checking in Jenkins.
Much of this data is worth visualizing in Kibana for analysis. Deciding what specific activities to log can be a bit involved. Areas that are a good idea to log include:
Diagnostics: For example, understanding the causes of repeated errors.
Auditing: Track how well errors in code have been fixed, and whether they succeed or fail a second time.
Profiling: For example, get an idea of how long it takes for certain parts of code to execute, for the purpose of identifying whether this has any impact on customer experience, and identify areas for improvement.
Statistics: Code performance, such as execution time or memory consumption.
Conclusion
Gaining complete visibility or your infrastructure will go a long way to make sure that your operation runs smoothly and that your CI/CD pipelines function at optimum efficiency. Open source tools remain an attractive option for DevOps because of its advantages and knowing how to get around drawbacks.
Finding the right tools is, of course, only the first step, but if you’ve picked good architectures and have implemented them properly, you are already moving in the right direction to a more streamlined operation.
Logz.io today at the AWS re:Invent 2019 conference launched Cloud Observability Platform, a monitoring and security cloud service based on ELK—open source Elasticsearch, a search engine; Logstash, a data processing pipeline; and Kibana, a set of visualization tools—and Grafana, an analytics and monitoring platform.
Company CEO Tomer Levy said it’s become apparent that developers have a strong preference for open source tools for managing cloud-native applications. As such, the adoption of ELK has increased substantially. With the increasing maturity of Grafana, it’s also becoming clear that organizations want to meld the open source analytics and monitoring tool with previous investments in ELK tools, he said.
However, not many IT organizations have a lot of ELK and Grafana expertise. To fill that gap, Logz.io is launching a managed cloud service based on Logz.io Log Management, an instance of ELK that the company has enhanced with machine learning algorithms and data analytics capabilities to enable DevOps teams to discover and resolve issues faster.
That core platform is being extended via Logz.io Infrastructure Monitoring, a beta release of a monitoring platform based on Grafana, and a security information and event management (SIEM) platform dubbed Cloud SIEM, based on an ELK stack that includes built-in threat detection tools.
In addition, Levy said Logz.io plans to add support for Jaeger, an open source distributed tracing tool being developed under the auspices of the Cloud Native Computing Foundation (CNCF).
Levy said it’s apparent to all that observability is a core tenet of DevOps, which he said is why there are already a half-million developers making use of ELK software. The challenge organizations are finding is that monitoring modern IT environments based on cloud-native applications is cumbersome and time-consuming. Logz.io is addressing that issue by providing an enterprise-grade instance of an open source monitoring platform that comes with hundreds of built-in connectors, which enables DevOps teams to employ a monitoring platform without having to be worried about becoming locked in to a specific vendor because they can switch over to open source code anytime they like, Levy noted.
Developers tend to favor open source tools because they don’t have to wait on purchasing approval to employ them. However, enterprise IT organizations tend to favor platforms that are supported by a commercial vendor. Logz.io is trying to address that issue by providing support for an open source platform that will make enterprise IT organizations comfortable enough to give up relying on a proprietary monitoring platform. Thus far, Logz.io has more than 700 customers, which the company expects to expand now that it has an instance of its platform that can be consumed as a cloud service.
It’s too early to say to what degree an open source platform might usurp proprietary platforms in the enterprise. Change tends to come slowly to enterprise IT organizations. However, as organizations embrace next-generation applications based on microservices and containers, enterprise IT organizations are clearly evaluating their monitoring platform options going forward.
This article covers important steps to monitor a typical Java application using Stagemonitor tool, Kibana as a dashboard viewing tool and Elasticsearch as a data processing engine.
Stagemonitor is an open source Java application performance monitoring (APM) tool that provides everything you need in development, quality assurance and production to monitor the performance of your Java web application.
Using this tool, we can have different dashboards such as JVM dashboards, application server, request metrics, host, request analysis, web analytics, etc.
Architecture of the Stagemonitor
Figure 1
The Stagemonitor agent sits inside the application and sends metrics and details about requests (request traces) to Elasticsearch. Metrics can optionally be sent to Graphite, Influx DB and Prometheus.
Prerequisites
Minimum of 2GB RAM and 200GB hard disk
Java version of 1.8 or more
Install latest versions of Kibana and Elasticsearch
Steps to Monitor Java App with Stagemonitor
Dev Time Monitoring With Browser Widget
This step illustrates how to enable a widget in dev time monitoring, adding the Stagemonitor plugin inside the Java app. We can do this in two ways: We can directly edit the pom.xml/build.gradle file by adding the required stage monitor plugins, or we can add dependencies in the application’s LIB directory.
Some of the important plugins we must have in pom.xml/build.grade to get monitoring dashboards are Stagemonitor configuration, Stagemonitor tracing Elasticsearch, Stagemonitor tracing, Stagemonitor-jdbc, Stagemonitor-jvm, Stagemonitor-os, Stagemonitor-web-servlet and Stagemonitor alerting.
After installing the agent, will have to create one Stagemonitor configuration file as stagemonitor.properties file under src/main/resources in case of maven application. Otherwise, the file must be located at the root of class path. At the end, it has to be under the WEB-INF/classes section.
The following is what the stagemonitor.properties file looks like:
stagemonitor.applicationName=My Application
stagemonitor.instanceName=dev
stagemonitor.instrument.include=org.example.app
Request Dashboard
In this section, we focus on setting up the time series database to collect the data and visualization tool to view the metrics in detail. Here we’ll use Elasticsearch as the time series database engine and Kibana as the visualization tool.
To get the data from Elasticsearch database, we will include properties in the stagemonitor.properties file:
After adding all the required values and restarting the server, we will be able to open the (automatically added) request analysis dashboard in Kibana. Along with that, we will be able to see different dashboards automatically.
To use Influx DB as a time series database, we’ll have to configure the values in stagemonitor.properties file:
We can track logs using Kibana/Stagemonitor. In the metrics dashboard, we can see matrices such as what URL address we’ve hit and the time to complete the request (Figure 2). We can track the host details such as how many servers our applications are running, the status of the server, memory stats, CPU stats, network stats, disk usage stats, networks processed, network packets errors, network packets dropped and how many cores of the CPU the host is using.
Stagemonitor provides all the required dashboards under the request analysis dashboard. To open the request analysis dashboard, click on: Dashboard – Load Saved Dashboard and select “Request Analysis.”
Figure 2
Figure 3 shows the details of the CPU, such as CPU usage and CPU load.
Figure 3
Below, Figure 4 shows total page views, unique number of visitors to the site and call tree, and how much time it takes to complete.
Figure 4
Figure 5 shows the transaction history and individual transaction details.
Figure 5
Figure 6 shows the JVM details and all the other memory matrices.
Figure 6
As mentioned above, we can monitor various application metrics and infrastructure metrics.
This article’s purpose is to compare the “big two” in the log analytics world—Splunk and the ELK Stack. But before we go into details, a short introduction to the competitors is required.
Splunk is the “Google for log files” heavyset enterprise tool that was the first log analysis software and has been the market leader ever since. The open-source ELK Stack of Elasticsearch, Logstash and Kibana is an up-and-coming rival that is a consolidated data analytics platform. The two compete in terms of features, usability and cost.
Overview
Splunk and the ELK Stack use two different approaches to solve the same problem. People typically choose one or the other based on how their organizations are structured and how much time they want to devote to log analytics. Splunk takes a pile of data and allows people to search through the information to extract what they need. ELK requires more work and planning in the beginning, but the value extraction is easier at the end.
The three key components of Splunk are its forwarder, which pushes data to remote indexers; indexer, which has roles for storing and indexing data and responding to search requests; and search head, which is the front end of the web interface where these three components can be combined or distributed over servers. Splunk also supports the integration of its functionalities in applications via SDKs. Common use cases include operational monitoring, security and user behavior analytics. Splunk is a paid service wherein billing is generated by indexing volume.
The ELK Stack is a set of three open-source products—Elasticsearch, Logstash and Kibana—all developed and maintained by Elastic. Elasticsearch is NoSQL database that uses the Lucene search engine. Logstash is a data processing and transportation pipeline used to populate Elasticsearch with the data (though also it supports other destinations including Graphite, Kafka, Nagios and RabbitMQ). Kibana is a dashboard that works on top of Elasticsearch and facilitates data analysis using visualizations and dashboards.
Both Splunk and the ELK Stack can be used to monitor and analyze infrastructure in IT operations as well as for application monitoring, security and business intelligence.
ELK vs. Splunk
Loading Data
Shipping data to Splunk is fairly easy. After installation, the forwarders come pre-configured for a wide selection of data sources such as files and directories, network events, windows sources and application logs, and they are used to import data into Splunk as shown:
In the ELK Stack, Logstash is used to ship data from the source to the destination. However, Logstash needs to be configured so that each field is identified before the data is shipped to Elasticsearch. This kind of configuration can be tricky for those who do not work with scripting languages (such as Bash, Python or Ruby), but there is good support online that can be found quite easily.
Visualizations
The Splunk web UI includes flexible controls that allow you to edit and add new components to your dashboard. Management and user controls can be configured differently for multiple users, with each having a customized dashboard. Splunk also supports visualizations on mobile devices with application and visualization components that are easy to customize using XML.
Kibana is the visualization tool in the ELK Stack, and like Splunk, the platform supports the creation of visualizations such as line charts, area arts and tables and the presentation of them in a dashboard. The search filter is always shown above the different views: If a query is used, it is automatically applied on elements of the dashboard. Splunk also has a similar option, but it involves configuration in XML. Still, Kibana does not support user management, but hosted ELK solutions provide it out of the box.
Search Capabilities
The search function is a key capability of any log management platform. Both the Splunk and ELK Stack’s web UIs support searching using a dedicated search field. The query syntax on Kibana is based on the Lucene query syntax while Splunk uses its own Splunk Search Processing Language (SPL). Those familiar with scripting languages may already be familiar with Lucene while SPL is proprietary and must be learned.
One other difference is that Splunk offers dynamic data exploration to help users to find and extract everything as a searchable field when formatted in a manner that allows searching for non-configured fields. Elasticsearch fields, on the other hand, need to be defined in advance to use aggregation over the log properties.
Here is one example of a query for each platform.
Kibana:
(beat.hostname: ES1 AND metricset.name: process) AND (system.process.username: root OR system.process.username: admin)
Splunk:
(index=* OR index=_*) (index=_audit) | search ( action=search NOT dmauditsearch ) "06:54"
The difference between SPL syntax and Lucene queries is that SPL supports the search pipeline (as in the example above) in which consecutive commands are chained together using a pipe character that allows the output of one command to be used as the input of the next one. Lucene query syntax is more straightforward and can work to generate the output from the query without additional transforming.
Traction and Community Support
Both Splunk and ELK Stack have large communities of users and supporters. ELK also has its own clear and extensive documentation for each separate tool, making it easy to get started. In addition, Elastic itself offers educational sessions worldwide.
In addition to having good documentation and a forum, Splunk, too, has customer and support platforms that offer various professional services. Splunk’s education program and instructors are available virtually or on site.
The Learning Curve
The ELK Stack’s learning curve is flat for what it’s meant to do. Elastic offers paid courses, but there is a lot of free material online due to the popularity of the open-source platform.
For Splunk, the learning curve is moderate in size, especially when carrying out more specialized analyzes. The company offers a trial period with extensive documentation, but the advanced Splunk educational courses are fairly expensive.
User Management
The ELK Stack provides role-based security as a separate paid tool. Splunk and managed-ELK services offer user management out of the box with user auditing included.
Pricing Levels
As already mentioned, Splunk is proprietary software with a price tag. After one integrates several data sources with the platform, the cost will increase greatly as data is continuously generated.
The open-source ELK Stack is free, but the true picture is not so black and white. The cost of the platform’s hardware and maintenance also adds up. To lower the cost of using ELK, features, plugins and tools must be developed.
Vendor Lock-In
Splunk’s high price tag comes with the benefit of offering an overall, well-rounded product. Users might be locked into a vendor, but that one vendor is all that is needed to do nearly anything. The open-source ELK Stack is seemingly free, but it does not provide many functionalities such as alerting out of the box—and it costs money to develop and maintain them.
One way to look at the Splunk-versus-ELK discussion is to frame it as the old Microsoft-versus-Linux debate. If you like Microsoft, you will probably prefer Splunk. If you like Linux, you will likely want to use the ELK Stack.
Final Thoughts
When one decides between these solutions, the ultimate choice should reflect not only on the platform but also on the customer’s specific needs.
Both Splunk and the ELK Stack are currently popular. But what might the future hold?
Splunk (Blue) vs. Elasticsearch (Red) on Google Trends
According to Google Trends, the ELK Stack has now overtaken Splunk in terms of the proportion of Google searches. But ELK’s traction does not stop there. As mentioned earlier, Splunk self-reports 12,000 total users. Elasticsearch is reportedly downloaded 500,000 times every single month. In IT departments, then, it is far more likely to meet people who are familiar with ELK than with Splunk, meaning that the adoption rate of the ELK Stack could “snowball” and increase even more in the future whenever ELK users join new companies or teams. People tend to use whatever software they already know or is already being used.
It’s clear that many functionalities are being added to the open-source ELK Stack. This, in turn, is shrinking the gap between it and Splunk. hose features currently found only in Splunk are likely to be added to ELK at some point in time.
So, if a more mature product is what you need, go with Splunk. But if you like a product that is more flexible, ELK is the way. Either way, it’ll be interesting to see how these two continue to compete.
