Tag: ELK

This article’s purpose is to compare the “big two” in the log analytics world—Splunk and the ELK Stack. But before we go into details, a short introduction to the competitors is required.

Splunk is the “Google for log files” heavyset enterprise tool that was the first log analysis software and has been the market leader ever since. The open-source ELK Stack of Elasticsearch, Logstash and Kibana is an up-and-coming rival that is a consolidated data analytics platform. The two compete in terms of features, usability and cost.

Overview

Splunk and the ELK Stack use two different approaches to solve the same problem. People typically choose one or the other based on how their organizations are structured and how much time they want to devote to log analytics. Splunk takes a pile of data and allows people to search through the information to extract what they need. ELK requires more work and planning in the beginning, but the value extraction is easier at the end.

The three key components of Splunk are its forwarder, which pushes data to remote indexers; indexer, which has roles for storing and indexing data and responding to search requests; and search head, which is the front end of the web interface where these three components can be combined or distributed over servers. Splunk also supports the integration of its functionalities in applications via SDKs. Common use cases include operational monitoring, security and user behavior analytics. Splunk is a paid service wherein billing is generated by indexing volume.

The ELK Stack is a set of three open-source products—Elasticsearch, Logstash and Kibana—all developed and maintained by Elastic. Elasticsearch is NoSQL database that uses the Lucene search engine. Logstash is a data processing and transportation pipeline used to populate Elasticsearch with the data (though also it supports other destinations including Graphite, Kafka, Nagios and RabbitMQ). Kibana is a dashboard that works on top of Elasticsearch and facilitates data analysis using visualizations and dashboards.

Both Splunk and the ELK Stack can be used to monitor and analyze infrastructure in IT operations as well as for application monitoring, security and business intelligence.

ELK vs. Splunk

Loading Data

Shipping data to Splunk is fairly easy. After installation, the forwarders come pre-configured for a wide selection of data sources such as files and directories, network events, windows sources and application logs, and they are used to import data into Splunk as shown:

In the ELK Stack, Logstash is used to ship data from the source to the destination. However, Logstash needs to be configured so that each field is identified before the data is shipped to Elasticsearch. This kind of configuration can be tricky for those who do not work with scripting languages (such as Bash, Python or Ruby), but there is good support online that can be found quite easily.

Visualizations

The Splunk web UI includes flexible controls that allow you to edit and add new components to your dashboard. Management and user controls can be configured differently for multiple users, with each having a customized dashboard. Splunk also supports visualizations on mobile devices with application and visualization components that are easy to customize using XML.

Kibana is the visualization tool in the ELK Stack, and like Splunk, the platform supports the creation of visualizations such as line charts, area arts and tables and the presentation of them in a dashboard. The search filter is always shown above the different views: If a query is used, it is automatically applied on elements of the dashboard. Splunk also has a similar option, but it involves configuration in XML. Still, Kibana does not support user management, but hosted ELK solutions provide it out of the box.

Search Capabilities

The search function is a key capability of any log management platform. Both the Splunk and ELK Stack’s web UIs support searching using a dedicated search field. The query syntax on Kibana is based on the Lucene query syntax while Splunk uses its own Splunk Search Processing Language (SPL). Those familiar with scripting languages may already be familiar with Lucene while SPL is proprietary and must be learned.

One other difference is that Splunk offers dynamic data exploration to help users to find and extract everything as a searchable field when formatted in a manner that allows searching for non-configured fields. Elasticsearch fields, on the other hand, need to be defined in advance to use aggregation over the log properties.

Here is one example of a query for each platform.

Kibana:

(beat.hostname: ES1 AND metricset.name: process) AND (system.process.username: root OR system.process.username: admin)

Splunk:

(index=* OR index=_*) (index=_audit)   | search ( action=search NOT dmauditsearch )  "06:54"

The difference between SPL syntax and Lucene queries is that SPL supports the search pipeline (as in the example above) in which consecutive commands are chained together using a pipe character that allows the output of one command to be used as the input of the next one. Lucene query syntax is more straightforward and can work to generate the output from the query without additional transforming.

Traction and Community Support

Both Splunk and ELK Stack have large communities of users and supporters. ELK also has its own clear and extensive documentation for each separate tool, making it easy to get started. In addition, Elastic itself offers educational sessions worldwide.

In addition to having good documentation and a forum, Splunk, too, has customer and support platforms that offer various professional services. Splunk’s education program and instructors are available virtually or on site.

The Learning Curve

The ELK Stack’s learning curve is flat for what it’s meant to do. Elastic offers paid courses, but there is a lot of free material online due to the popularity of the open-source platform.

For Splunk, the learning curve is moderate in size, especially when carrying out more specialized analyzes. The company offers a trial period with extensive documentation, but the advanced Splunk educational courses are fairly expensive.

User Management

The ELK Stack provides role-based security as a separate paid tool. Splunk and managed-ELK services offer user management out of the box with user auditing included.

Pricing Levels

As already mentioned, Splunk is proprietary software with a price tag. After one integrates several data sources with the platform, the cost will increase greatly as data is continuously generated.

The open-source ELK Stack is free, but the true picture is not so black and white. The cost of the platform’s hardware and maintenance also adds up. To lower the cost of using ELK, features, plugins and tools must be developed.

Vendor Lock-In

Splunk’s high price tag comes with the benefit of offering an overall, well-rounded product. Users might be locked into a vendor, but that one vendor is all that is needed to do nearly anything. The open-source ELK Stack is seemingly free, but it does not provide many functionalities such as alerting out of the box—and it costs money to develop and maintain them.

Spunk is Splunk. There are many varieties of ELK:

• The open-source ELK Stack platform (Elastic)
• Hosted Elasticsearch (AWS)
• AI-powered ELK on an enterprise-grade platform (Logz.io)

One way to look at the Splunk-versus-ELK discussion is to frame it as the old Microsoft-versus-Linux debate. If you like Microsoft, you will probably prefer Splunk. If you like Linux, you will likely want to use the ELK Stack.

Final Thoughts

When one decides between these solutions, the ultimate choice should reflect not only on the platform but also on the customer’s specific needs.

Both Splunk and the ELK Stack are currently popular. But what might the future hold?

According to Google Trends, the ELK Stack has now overtaken Splunk in terms of the proportion of Google searches. But ELK’s traction does not stop there. As mentioned earlier, Splunk self-reports 12,000 total users. Elasticsearch is reportedly downloaded 500,000 times every single month. In IT departments, then, it is far more likely to meet people who are familiar with ELK than with Splunk, meaning that the adoption rate of the ELK Stack could “snowball” and increase even more in the future whenever ELK users join new companies or teams. People tend to use whatever software they already know or is already being used.

It’s clear that many functionalities are being added to the open-source ELK Stack. This, in turn, is shrinking the gap between it and Splunk. hose features currently found only in Splunk are likely to be added to ELK at some point in time.

So, if a more mature product is what you need, go with Splunk. But if you like a product that is more flexible, ELK is the way. Either way, it’ll be interesting to see how these two continue to compete.

— Asaf Yigal